Every LLM feature ships with a hidden number: the total milliseconds a user will wait before the feature feels broken. A latency budget makes that number explicit, splits it across pipeline stages, and attaches an SLO with an error budget to keep it honest. Latency budgets are arithmetic, not opinions: fix the user-visible target, subtract the stages you cannot compress, and hold what remains accountable with time-to-first-token (TTFT) and end-to-end percentiles. Users judge speed by the first visible token, not the last, and providers designed their APIs around that: Google’s Gemini API pairs generateContent with streamGenerateContent, an endpoint that returns the model response as a stream. This worksheet turns that fact into targets each team can own.
Where the milliseconds go
Break the request path into stages and give each one a metric and an owner before arguing about targets. A RAG chain has more hops than most teams expect: in the reference RAG flow documented by Microsoft, the orchestrator queries the search index, packages the top results plus the query into a prompt, and only then calls the language model. Every hop — client network, embedding, vector search, prompt assembly, model queueing — spends budget before a single output token exists. Bedrock shows how much control exists at the API level: Amazon Bedrock ships a dedicated streaming operation, InvokeModelWithResponseStream, and a per-request performance setting where the latency profile can be set to standard or optimized. Check the responseStreamingSupported flag per model before promising streaming targets, and measure each stage separately so a slow retrieval cannot hide behind a fast model.
Levers that cut latency
Once stages are measured, three levers move real numbers: shrink the prompt, cache what repeats, and route easy traffic to a smaller model. Caching is the cheapest first win on repeated context. On Google Cloud, implicit context caching is enabled by default for all projects and cuts the price of cached input tokens by 90%, and the response metadata exposes cachedContentTokenCount so you can verify cache hit rate in production instead of hoping. Put stable instructions and long reference text at the front of the prompt where hits are likeliest, keep variable content at the end, and treat a falling hit rate as a latency regression even while the SLO is still green. For model routing, record in the worksheet which share of traffic is eligible for the small model and which quality gate protects it.
The SLO worksheet itself
Build the worksheet as one row per stage, one owner per row, and one percentile that matches how users experience the feature. Work through it in this order:
- Fix the user-visible target first: the end-to-end budget for the whole feature, stated as a percentile over a rolling window.
- List every stage on the path, from client request to last token, and assign each an owner who can change its code.
- Allocate the budget across stages, leaving explicit slack for retries and bursts.
- Instrument each stage so TTFT and per-stage spans are logged per request, not sampled away.
- Define what happens when the budget is missed — page, throttle, or degrade — in writing, before the first violation.
The allocation below is an illustrative example for a hypothetical chat feature with a 1,500 ms end-to-end target; replace every share with your own measured spans before adopting it.
| Stage | Illustrative share | Metric | Owner |
|---|---|---|---|
| Retrieval (embedding + search) | 30% | p95 span | Search team |
| Prompt assembly + cache lookup | 10% | p95 span | App team |
| TTFT (queue + prefill) | 35% | p95 TTFT | Platform team |
| Stream to last token | 25% | p95 completion | Model serving |
Set targets with error budgets
A latency target only becomes an SLO when missing it has a consequence. The Google SRE workbook is blunt about this: 100% reliability is the wrong target, because each extra nine costs more while its marginal utility approaches zero. The same discipline applies to latency: pick a percentile users feel, attach an error budget that states how many misses per window are tolerated, and agree in advance what burns when the budget is gone — feature freeze, slower rollouts, or a paging rotation. State the window and the percentile in the same sentence as the number, or reviews will argue about both. Targets without a budget become wallpaper within a quarter.
Spend the budget deliberately
When a stage overruns, there are three honest options: compress it, cache it, or move budget from a stage with slack — never quietly widen the end-to-end target. Two adjacent practices keep the worksheet alive. First, per-stage logging decides whether a miss is diagnosable, so align it with your observability decisions about what to log and redact. Second, bursts destroy budgets that averages hide, so enforce upstream concurrency with the same care you put into gateway rate limiting and quota design. Revisit the allocation after any model swap, provider change, or prompt-size change, because each of them silently re-slices the budget.