Week 2 · Production Python · about 3 minutes
What you will be able to do
Call an LLM from a customer network without retrying forever or retrying client errors.
The idea
Retry only when another attempt could succeed: 429 and most 5xx. Read Retry-After when the gateway sends it. Otherwise wait with exponential backoff plus a little jitter so every replica does not wake up together.
Never retry a 400 or a 401. Those are your bug or your credential, and a loop makes them an incident. Cap the attempts. When the cap is hit, fail the request with a log line that includes the status and the attempt count.
delay = 0.5
for attempt in range(5):
response = httpx.post(URL, json=payload, timeout=20)
if response.status_code == 200:
return response.json()["text"]
if response.status_code in (429, 500, 502, 503):
retry_after = response.headers.get("Retry-After", "")
# Seconds, or an HTTP date; fall back to backoff for a date.
wait = float(retry_after) if retry_after.isdigit() else delay
time.sleep(wait + random.random() * 0.3)
delay = min(delay * 2, 8)
continue
response.raise_for_status()
raise RuntimeError("LLM gateway exhausted retries")A retry without a cap is an outage you shipped yourself.
Key takeaways
- Retry 429 and 5xx only, with a hard attempt cap.
- Honor Retry-After, otherwise exponential backoff with jitter.
- Log status and attempt count when you give up.
Part of the Forward Deployed Engineer roadmap. New to the role? Start with what a Forward Deployed Engineer does.