AI gateway rate limiting works when the gateway meters tokens, not just requests. Every major provider enforces a different mix of tokens-per-minute, requests-per-minute, daily and spend ceilings, and each one scopes those ceilings differently, so a proxy that only counts calls will pass its own limits and still fail upstream. Tokens per minute and requests per minute limits in Azure OpenAI are defined per region, per subscription, and per model or deployment type, which means your usable capacity is fragmented across regions and deployments before any internal team has made a single call. A gateway that models those scopes — per key, per model, per region — is the only design that degrades predictably under load.
Why plain request limits fail
One LLM call can consume 50 tokens or 50,000. A limit of 200 requests per minute tells you nothing about whether a burst of long-context summarisation jobs will fit inside the provider quota that actually binds. The failure mode is quiet: interactive chat keeps working while the nightly ingestion batch exhausts the account-level pool, and the on-call engineer inherits a problem created at design time. Sizing an AI gateway has the same shape as GPU capacity planning for LLM inference: you size for the token stream, not the call count, and you plan for the burst, not the average.
How providers meter quota
The three scopes you must model differ at each provider. Azure OpenAI pools tokens per minute and requests per minute per region, subscription and model; for example, a Tier 1 GlobalStandard gpt-4o-mini deployment is listed at 20,000 RPM and 2,000,000 TPM per region, and Azure also applies monthly usage tiers per model. Amazon Bedrock takes a different unit: the bedrock-runtime endpoint counts input and output tokens together against a single per-model quota, and the default daily quota is the per-minute quota multiplied by 24 × 60. Gemini adds dimensions the other two do not expose: Gemini API rate limits are applied per project, not per API key, and requests-per-day quotas reset at midnight Pacific time.
| Provider | Metered units | Quota scope | Extra ceiling |
|---|---|---|---|
| Azure OpenAI | TPM and RPM | Region, subscription, model | Monthly usage tiers per model |
| Amazon Bedrock | TPM (input plus output); RPM for some models | Model, Region | Daily token quota by default |
| Gemini API | RPM, TPM, RPD | Project | Spend limit per rolling window |
Two consequences follow. First, multi-region failover does not just add capacity — it multiplies quota pools, but only if the gateway tracks them separately. Second, Azure documents that you might receive 429 responses even when token usage metrics appear below quota, so utilisation dashboards alone are not a contract; headroom below your own quota is part of the design, not slack.
Designing the gateway token bucket
The core primitive is a per-key token bucket with a per-model global pool behind it. The gateway should reserve before dispatch and refund after, using the usage object the provider returns, so partial consumption never leaks budget. A workable request path, stated as a design pattern rather than a measured benchmark:
- Estimate the call cost: measured prompt tokens plus the reserved maximum output tokens.
- Reserve that estimate against the per-key bucket and the shared per-model pool; queue or reject if either is short.
- After the response, read actual usage from the provider payload and refund the difference between reserved and consumed tokens.
- On a 429, honour the provider’s retry timing, apply exponential backoff with jitter, and mark the pool degraded so new requests queue at the gateway instead of hammering the provider.
- Split interactive and batch traffic into separate budgets so a nightly job can never starve user-facing latency.
Refund-on-completion matters more than precision of the estimate: a gateway that only debits after the fact offers no admission control, while one that never refunds grows steadily conservative as estimate errors accumulate.
Mapping quota units to budgets
Token quotas are a cost control whether you intend them to be or not. Gemini additionally enforces spend-based rate limits evaluated on a rolling 10-minute window: $10 for Tier 1, $50 for Tier 2 and $200 for Tier 3 — spend is itself a rate limit, priced in currency instead of tokens. For gateway design this means the budget layer cannot be a monthly afterthought: a single expensive request can trip a spend ceiling in minutes. Convert each team’s monthly budget into a token-per-minute allowance per key, and treat retrieval workloads carefully, because RAG ingestion over managed vector databases front-loads token spend in bursts that look nothing like steady chat traffic.
A workable rollout checklist
- Inventory every provider scope you depend on: region pools, per-model quotas, project-level limits and any spend ceiling.
- Pick the per-key dimension that matches your organisational boundary — team, product or customer — and stick to one.
- Implement reserve-and-refund token accounting with actual usage from provider responses.
- Alert on utilisation thresholds and refund drift, not only on 429 spikes.
- Load-test to the provider’s documented limits in staging before each capacity change.
- Document the retry, queueing and degradation policy so client teams can design against it.
Gateways that follow this checklist fail in the queue, where you control the outcome, instead of at the provider, where you do not.
Sources
- Azure OpenAI in Microsoft Foundry — quotas and limits (Microsoft Learn)
- Quotas for the Amazon Bedrock bedrock-runtime endpoint (AWS documentation)
- Gemini API rate limits (Google developers documentation)