Six Attempts: Bounded LLM API Retry Budgets That Hold

When a call to a hosted model fails, the outage risk is rarely the error itself — it is a client that retries the wrong errors or retries them forever. Vendor guidance for Amazon Bedrock is concrete: honor Retry-After when the service sends it, retry only transient throttling and capacity errors, and cap the whole operation at a bounded LLM API retry budget of six total attempts — the initial request plus up to five retries. Client-side failures such as 400 validation errors, 401 authentication problems and 404 missing deployments should fail fast: retrying them burns latency and, on token-metered quotas, keeps consuming capacity that still counts against your limits.

Classify the Error First

Retries are only safe once you know which failure class you are handling. The Bedrock scaling guidance separates admission from capacity: a 529 overloaded_error signals insufficient serving capacity right now, while a 429 ThrottlingException means you exceeded an account quota. The same document warns that some runtime operations also return HTTP 429 for ModelNotReadyException, so inspect the API-specific error type rather than the status code alone. On the Azure side the mapping is similar: follow the retry-after-ms header when Azure OpenAI returns 429, and retry 500 and 503 with exponential backoff.

Status	Bedrock meaning	Azure OpenAI meaning	Action
429	ThrottlingException — quota or rate limit exceeded	Quota, capacity throttling or temporary limit adjustment	Honor the retry hint, back off, then review quotas
500 / 503	Server error or temporary capacity constraint	Transient service issue, backend overload	Retry with exponential backoff
529	overloaded_error — model cannot serve the request now	Not used	Treat as transient capacity; consider another region
400 / 401 / 404	Validation, credentials or missing resource	Same classes	Do not retry; fix the request

Set a Bounded Retry Budget

An unbounded retry loop converts a provider hiccup into a self-inflicted denial of service. The AWS worked example limits an operation to the ceiling quoted above — the initial call plus retries — and shows how it maps to SDK settings, because botocore’s total_max_attempts counts the initial request, while the OpenAI and Anthropic SDKs’ max_retries counts only retries. The same numeric budget therefore needs different constants per client library:

import boto3
from botocore.config import Config

config = Config(retries={"total_max_attempts": 6, "mode": "standard"})
client = boto3.client("bedrock-runtime", config=config)

With an OpenAI-compatible endpoint the equivalent setting is max_retries=5. Budget size is a product decision: the documented six-attempt ceiling with a one-second base delay and capped growth fits most interactive latency targets, while asynchronous pipelines — like the reindex jobs described in the embedding model migration runbook — can trade longer waits for fewer attempts to avoid duplicate-token cost.

Tune Backoff, Jitter, Caps

The Bedrock recommendation reduces to four levers, applied in order:

Start with a short delay — one second in the documented example.
Multiply the delay after each attempt and cap the maximum delay inside your latency budget.
Add random jitter so concurrent workers never retry in lockstep.
Honor a Retry-After header whenever one is present, adding jitter on top.

AWS calls out the failure mode explicitly: avoid immediate or synchronized retries from multiple clients, because they increase load and delay recovery. Exponential backoff without jitter is just a thundering herd on a schedule.

Split Timeouts From Retries

Configure connection and read timeouts separately from retries, based on the documented maximum inference duration for your model and operation. A read timeout shorter than a valid long-running inference request causes avoidable retries and duplicate work: you re-send a prompt that is still being processed, and pay for both attempts. There is also a quieter failure mode documented for AWS networking: NAT Gateways and interface VPC endpoints drop idle TCP connections after 350 seconds, silently, so a pooled connection can already be dead before your next request reuses it. The fix has two parts that only work together — enable TCP keep-alive in the SDK client (tcp_keepalive=True in boto3) and lower the kernel’s net.ipv4.tcp_keepalive_time below the idle window. The Linux default of 7200 seconds never fires in time; the documented example of 45 seconds does.

Ramp Instead of Bursting

Sustained 503 or 529 responses mean retries alone will amplify the problem. The documented response is to stop the ramp: return to the last stable request rate, bound client-side concurrency, queue work, and defer or shed lower-priority requests until capacity recovers. When you raise traffic again, change one load dimension at a time and watch success rate, throttling, latency percentiles and queue depth at each step before the next increase. Plan in tokens, not requests: on the bedrock-mantle endpoint, admission counts input tokens plus the requested max_tokens, so oversized limits quietly shrink effective throughput — the same token accounting that drives the provider differences in the prompt caching cost comparison. For large offline jobs, batch inference exists precisely so you never need interactive retry behavior at scale.