Cloud AI

Batch processing is the cheapest single lever in LLM inference cost engineering: OpenAI’s Batch API and Anthropic’s Message Batches API both charge 50% less than synchronous endpoints, with completion windows measured in hours rather than seconds. For cloud teams in Portugal and the EU running evaluations, classification pipelines, embedding generation, or bulk content processing, moving latency-tolerant traffic to a batch path halves token spend without changing models or prompts. The trade-off is a different operational contract — strict input limits, expiry semantics, and asynchronous failure handling. This guide covers what each LLM batch API enforces and how to route workloads onto it safely.

How Batch APIs Cut Costs

Both providers implement the same core pattern. You collect many requests into a single job — OpenAI as a JSONL file uploaded through the Files API, Anthropic as a list of request objects — and the platform executes them asynchronously against a separate pool of rate limits. Each request carries a unique custom_id you use to map outputs back to inputs, because the output order is not guaranteed to match the input order. When the job completes, you download one results file per batch; failed requests land in a separate error file rather than aborting the whole job.

The economics matter more than the mechanics. A nightly evaluation or backfill job does not need second-level latency, so paying synchronous rates for it is pure waste. The table below summarizes the operational differences that drive routing decisions.

OpenAI Batch API Limits

The per-batch ceilings are the first constraint to plan around. A single OpenAI batch may include up to 50,000 requests, and the input file can be up to 200 MB in size. Batch creation itself is throttled at 2,000 batches per hour, and each model has a cap on enqueued prompt tokens visible in Platform Settings. Batches that do not finish inside the 24-hour window move to an expired state: unfinished requests are cancelled, completed responses remain available, and tokens consumed by completed requests are still billed.

The submission flow is deterministic and easy to automate:

1. Build a JSONL file where each line is one request with a unique custom_id, targeting a single model and one supported endpoint such as /v1/chat/completions or /v1/embeddings.
2. Upload the file with the Files API using purpose="batch".
3. Create the batch referencing the file ID and the 24-hour completion window.
4. Poll the batch status; on completed, download the output file and reconcile results by custom_id.
5. Handle the error file separately — expired and failed requests carry structured error codes, not exceptions.

Anthropic Message Batches Constraints

Anthropic caps a Message Batch at either 100,000 requests or 256 MB, whichever is reached first. Most batches complete within an hour, but results become available only when all messages finish or after 24 hours, and batches that exceed the window expire with unfinished requests cancelled. Results stay downloadable for 29 days, after which the batch remains visible but its payloads are gone — a retention boundary that matters for audit pipelines in regulated EU environments.

Almost every Messages API feature is batchable, including vision, tool use, and extended thinking, but a small parameter set is rejected outright: stream: true, Fast mode, Threads-related parameters, and max_tokens: 0 cache pre-warming. Because batched requests execute concurrently and in arbitrary order, prompt caching inside batches is best-effort; if your jobs share long static prefixes, combining batching with the prefix-reuse techniques in our prompt caching engineering guide stacks the savings, since the caching discount applies to tokens and the batch discount applies to the request price.

Routing Workloads to Batches

The routing decision is about latency tolerance, not volume. Interactive chat, live tool-calling agents, and anything user-facing must stay on synchronous endpoints. Batch paths fit three shapes: scheduled work (nightly evaluations, dataset labelling, embedding rebuilds), bulk generation (product descriptions, summaries, moderation sweeps), and bursty jobs that would otherwise collide with per-model rate limits, since the batch pool is billed and throttled separately. A useful heuristic: if the consumer of the result is a cron job or a queue rather than a human, it belongs on a batch API.

Failure handling changes too. In synchronous code, a 429 triggers a retry with backoff; in batch code, a rejected request is a row in an error file that your orchestrator must reconcile by custom_id. Design the pipeline around idempotent retries keyed on those identifiers, and monitor batch state transitions rather than HTTP status codes. Teams operating under the EU AI Act should also note that batch outputs persist on provider infrastructure for up to 29–30 days, which interacts with the retention and transparency duties summarised in our Article 53 compliance analysis.

A Batch Migration Checklist

• Inventory every LLM call site and tag it synchronous-interactive or latency-tolerant.
• Estimate monthly token volume per tolerant workload and project the 50% saving against engineering effort.
• Confirm target models support the batch path on your chosen provider.
• Split inputs into jobs that stay under the request-count and size ceilings, with headroom for retries.
• Strip unsupported parameters (streaming, cache pre-warming) before submission.
• Implement custom_id-keyed reconciliation of output and error files into your data store.
• Download and archive results before the 29–30 day retention window closes.
• Alert on expired or failed request counts per batch, not just on job-level status.

Sources

• OpenAI — Batch API documentation
• Anthropic — Message Batches API documentation