A slow LLM feature is almost never uniformly slow, and treating it as one number guarantees a bad diagnosis. Time to first token (TTFT) is how long the user waits before the first word appears; inter-token latency (ITL) is the cadence of every word after that. They have different causes, different fixes and, on managed platforms, different owners, so splitting LLM latency metrics is the first diagnostic step, not the last. On Amazon Bedrock the split is explicit: Bedrock’s own documentation states the identity: InvocationLatency (ms) = TimeToFirstToken (ms) + (OutputTokenCount / OTPS) * 1000, where OTPS is output tokens per second. Read it as a decision rule: if TTFT is high but OTPS is stable, your problem is input length or queueing before generation starts; if TTFT is fine but OTPS dropped, the model host is generating tokens more slowly — a service-side throughput change that deserves an alarm.
Why wall-clock latency misleads
Amazon Bedrock’s InvocationLatency metric reports the wall-clock time of an inference request, from when the request is received to when the last output token is produced. The same elevated value can hide two completely different conditions: the model generating tokens more slowly (a service-side throughput change you should escalate), or the model generating more tokens per request (a workload change such as a longer prompt, an updated system prompt, or a model update that produces longer responses). Alarming on total latency alone fires on both, which is why teams get paged for incidents that are really prompt regressions. The remedy is to keep the raw metric for capacity planning but never let it page anyone on its own — decompose it first, and apply the same envelope-only telemetry discipline you would use for LLM observability logging so the diagnosis does not require raw prompts.
| Signal | Measures | Scales with | Act when |
|---|---|---|---|
| TimeToFirstToken | Wait before the first output token | Input prompt length, queueing | Rises while OTPS holds steady |
| OTPS | Decode throughput of the model host | Model version and host load | Drops below a threshold set from your baseline |
| InvocationLatency | Total wall-clock per request | Both of the above combined | Never alone — decompose before acting |
Prefill drives first-token latency
An inference request passes through two compute-bound stages on the model host. In prefill, the model processes the entire input prompt in a single forward pass and produces the first output token; the duration of this stage scales primarily with input length and is the main driver of TimeToFirstToken. This is why the fastest TTFT fix is rarely infrastructure: trimming a bloated system prompt, cutting retrieved context that the answer never uses, or reusing cached prompt prefixes moves TTFT directly, because every removed input token is work the prefill pass no longer does. When you see TTFT creep upward week over week while decode stays stable, audit what accumulated in the prompt — retrieved chunks, few-shot examples, instruction bloat — before you touch quotas or regions.
Decode sets the throughput floor
In decode, the model generates each subsequent output token sequentially, one token per forward pass, so the total time of this stage scales with the number of output tokens. Per-token decode time is fairly stable for a given model and host load, which is exactly what makes OTPS a useful throughput signal: it isolates the service-side component by dividing output tokens by the time that remains after TTFT. A stable OTPS over time means the model is generating at expected throughput even when total latency is elevated because prompts or responses grew. A drop in OTPS is the model-side change you actually want to be paged for. Treat the output side honestly too: if a model update started producing longer responses, that is a cost and UX problem worth its own review, not a throughput incident.
Split the metrics in CloudWatch
There is a practical catch before you start: Amazon Bedrock publishes TimeToFirstToken only for the streaming API operations ConverseStream and InvokeModelWithResponseStream, so this whole diagnostic path applies solely to traffic on streaming endpoints. With that confirmed, the decomposition is a metric math expression over published CloudWatch metrics, not custom instrumentation:
- In the CloudWatch console, open Metrics, All metrics, search for Bedrock and select the By ModelId dimension.
- Select InvocationLatency, OutputTokenCount and TimeToFirstToken for the model you monitor.
- Set Statistic to p50 and Period to 5 minutes for all three metrics.
- Add a math expression computing OutputTokenCount divided by (InvocationLatency minus TimeToFirstToken), multiplied by 1000, and label it OTPS.
- Alarm on OTPS with a static threshold around 80 percent of your measured baseline, evaluating 3 of 5 datapoints, or use anomaly detection once you have at least two weeks of history.
The p50 view keeps the alarm about the median user experience; keep a p95 graph beside it for tail investigations without letting the tail page the on-call engineer.
Tuning the trade-off
Once you can see both stages, you can trade them deliberately. On self-hosted serving, By default the vLLM scheduler prioritizes prefills and never batches prefill and decode together, a policy that optimizes TTFT while accepting slower inter-token latency and less efficient GPU utilization. Enabling chunked prefill reverses the priority — decode requests are batched first, pending prefills are chunked into whatever token budget remains — which improves inter-token latency and GPU utilization at the cost of a longer wait for the first token. Choose by product shape: a chat UI lives on TTFT and should keep the default, while a background summarization job that streams progress to a progress bar cares more about steady cadence. The same reasoning applies to the self-hosting decision itself, which for some workloads needs far less hardware than teams assume, as covered in our guide to embeddings on CPU.
On managed platforms the lever is different. Amazon Bedrock’s latency-optimized inference covers Amazon Nova Pro, Claude 3.5 Haiku and Llama 3.1 405B and 70B models, with reduced response times available by setting the latency parameter to optimized through the performanceConfig field on the runtime API, with no fine-tuning or extra setup. Verify two operational details before you rely on it: requests beyond the usage quota fall back to standard inference and standard rates, and per-request routing is visible in the API response and CloudTrail, so confirm what you actually got rather than what you asked for. Whichever side you tune, fold the endpoints into your regular review cadence — an audit checklist in the style of our MCP server security walkthrough keeps config drift from silently undoing the latency profile you shipped.
Sources
- AWS — Diagnose InvocationLatency increases using output tokens per second (OTPS), consulted 2026-10-19.
- AWS — Optimize model inference for latency (Amazon Bedrock), consulted 2026-10-19.
- vLLM — Optimization and Tuning (v0.8.5), consulted 2026-10-19.