Batch inference APIs sell you the same models at half price in exchange for latency you can budget for: results arrive in minutes or hours instead of milliseconds. Anthropic charges all batch usage at 50% of its standard synchronous API prices, and Amazon Bedrock bills batch inference at a 50% lower price than the same models on demand. That discount covers evaluation runs, dataset backfills, document enrichment and nightly summarisation — any workload where nobody is waiting on the response. Interactive traffic stays exactly where it is.
What You Trade for Half-Price Tokens
A batch API is an asynchronous job queue with a pricing incentive attached. You submit many independent requests at once, the provider processes them off the interactive path, and you collect results afterwards. Every record is handled on its own, with no conversation state flowing between records — that independence is precisely what makes the capacity cheap for the provider to schedule in low-demand windows.
The trade is explicit: you give up per-request latency guarantees and streaming, and you accept a completion window measured in minutes or hours. This is a different lever from tuning the synchronous path. If parts of your product still need live responses, measuring time-to-first-token separately from inter-token latency keeps that residual online traffic honest — the TTFT vs ITL split remains the reference. Batch absorbs everything else.
Provider Mechanics Compared
The three major implementations differ in where data lives and how completion is signalled. On AWS, you upload input files to an S3 bucket, submit a job, and retrieve the output files from S3 when the job completes. On Google’s platform, batch jobs on Google’s platform are asynchronous requests made directly against the model resource, with no endpoint deployed underneath, so you are not paying for idle serving infrastructure while work waits. Anthropic batches travel inside the API request itself, with results fetched by polling the batch object.
| Provider | Submission | Data path | Completion signal | Discount |
|---|---|---|---|---|
| Anthropic | Message Batches API | Requests inside the batch object; results downloadable | Most batches finish in less than 1 hour; poll for status | All usage at 50% of standard prices |
| Amazon Bedrock | Batch inference job over the runtime API | Input JSONL files in Amazon S3; output written back to S3 | Polling, or Amazon EventBridge notifications on state change | 50% lower than on-demand for supported models |
| Google Vertex AI | BatchPredictionJob against the Model resource | Accumulated data processed in a single request | Asynchronous job completion; no endpoint to manage | Model-specific batch rates |
At published rates, Claude Sonnet 5.5 batch usage lists at $1 per million input tokens and $5 per million output tokens; the same page states all batch usage is charged at half of standard prices, so the synchronous equivalents are double those figures.
Limits That Bite in Production
Anthropic caps both scale and time: one Message Batch stops at 100,000 requests or 256 MB, whichever limit it hits first, batches expire if processing does not complete within 24 hours, and results remain downloadable for 29 days after creation. Streaming is unavailable in batch mode, and each batched request needs a max_tokens value of at least 1. Anthropic also warns that high-throughput batch processing can push a workspace slightly past its configured spend limit, so set billing alerts before your first large job.
On the AWS side, Bedrock batch inference does not support tool calling or structured output. Each record in the input JSONL file is processed independently without multi-turn interaction, and batch inference is not offered for provisioned models. Workloads that depend on agentic tool use stay synchronous — and that agent infrastructure still deserves a hardening pass, starting with an MCP server audit checklist.
Google applies batch semantics to the Model resource rather than a deployed Endpoint, so in practice any feature that assumes a live endpoint session needs checking against the batch path before you migrate, not after.
A Migration Checklist
- Classify calls by SLA: anything a user waits on stays synchronous; everything else is a batch candidate.
- Make records independent: strip multi-turn state, tool dependencies and response-format expectations from each item before it enters the batch.
- Assign a unique custom ID to every record so outputs join back to inputs without guesswork.
- Move storage first: for Bedrock, get the input file into S3 and the service role permissions right before writing job code.
- Replace polling with events where available — EventBridge reports Bedrock job state changes so your orchestrator is not looping on describe calls.
- Reconcile after every job: count succeeded, failed and expired records, and archive raw outputs before the provider retention window closes.
- Measure realized savings per million tokens against your own synchronous baseline for the same period, not against list prices.
Cost Edge Cases Worth Checking
Batch discounts stack awkwardly with other optimisations. Prompt caching cuts the cost of repeated prefixes in synchronous traffic; Anthropic notes that batches with shared context can benefit from a longer cache duration, but a zero-token cache pre-warm request is rejected inside batches because the ephemeral cache entry would likely expire before any follow-up request runs. If your batch shares a large static prefix across records, weigh cache hits inside the batch against simply paying the halved rate on cold input.
Watch the fleet boundary too. Teams that mirror every synchronous call into batch mode for offline evaluation double their token volume even at half price — the offline signal has to justify the second bill on its own. The clean pattern is a one-off backfill followed by incremental nightly batches sized from real throughput, reviewed each month against the invoice.