Batch API

A surprising amount of LLM work is not interactive. Generating embeddings for a million documents, running a nightly eval suite, classifying a two-year backlog of support tickets, summarizing yesterday's call transcripts: none of it needs an answer this second. The Batch API exists for exactly that work. You hand the provider a large pile of requests, it processes them on its own schedule when capacity is free, and you collect the results later, for half the price. It is the single easiest large cost win in LLM engineering, because it requires no model change, no prompt change, and no quality loss.

Sync vs async: two different contracts#

A normal synchronous call is a promise of immediacy. You send one request, the provider reserves capacity to serve it right now, tokens stream back within seconds, and you pay full price for that reservation. The Batch API flips the contract. You bundle requests into a single job, each tagged with a custom ID so you can match results back, submit it, and walk away. The provider slots the work into idle GPU time between real-time traffic peaks instead of holding capacity open for you, and passes the savings back as a discount.

The mechanics are similar everywhere. With OpenAI you upload a JSONL file, one request per line, and create a batch job pointing at it. With Anthropic you POST an array of requests directly to the Message Batches endpoint. Either way you then poll the job status (or register a webhook), and when it flips to ended you download a results file. Results arrive in arbitrary order, which is why the custom ID on each request matters: it is the only reliable join key. Individual requests within a batch succeed or fail independently, so a handful of malformed requests do not sink the other 99,000.

The numbers: 50% off, 24-hour ceiling#

The headline figure is remarkably consistent across the industry as of mid-2026: a flat 50% discount on both input and output tokens versus synchronous pricing. Anthropic's Message Batches API, OpenAI's Batch API, and Google's Gemini Batch API all land on exactly this number, and all three pair it with the same service guarantee: results within 24 hours. In practice the ceiling is rarely hit. Anthropic states most batches finish in under an hour; OpenAI's typically complete in one to six hours depending on load.

Scale limits are generous. Anthropic accepts up to 100,000 requests or 256 MB per batch, with results retrievable for 29 days. OpenAI caps a batch at 50,000 requests and a 200 MB input file. A worked example makes the economics concrete: classifying 1 million support tickets with Claude Haiku 4.5 at roughly 600 input and 80 output tokens each is 600M input tokens ($600 at $1 per million) plus 80M output tokens ($400 at $5 per million), so $1,000 synchronous. The same job batched costs $500. At Opus or GPT-5.x prices the absolute savings scale into the tens of thousands of dollars for recurring pipelines.

The quiet second discount: rate limits and cache stacking#

Two less-advertised properties often matter as much as the price. First, batch jobs draw from a separate, much larger rate-limit pool. OpenAI's batch requests do not consume your standard per-model token limits at all, and Anthropic's tier 1 organizations can hold 100,000 requests in the processing queue. That means you can run a million-item backfill overnight without throttling the production traffic your users depend on, something that is nearly impossible to do politely through the synchronous API.

Second, the discount stacks with prompt caching. If every request in your batch shares the same long system prompt or document context, cache reads inside the batch are billed at the cached rate and then halved again. On Anthropic the combination can push the effective input price down by as much as 95% versus an uncached synchronous call. For bulk extraction jobs where a 5,000-token instruction block repeats across 50,000 documents, structuring the prompt for cache hits before batching it is the difference between a real bill and a rounding error.

Where batch fits#

Good batch candidates share one trait: nobody is waiting. The canonical fits are evals and regression suites, where you re-run hundreds or thousands of test cases against a new prompt or model and only care about the aggregate report; backfills, where a new classifier or summarizer must be applied to years of historical records; bulk extraction and enrichment, pulling structured fields out of contracts, resumes, or product listings; embeddings generation over large corpora; synthetic data generation for fine-tuning; and content moderation sweeps over user-generated archives. Recurring nightly or weekly jobs are especially good fits because the 24-hour window slots naturally into a cron schedule: submit at midnight, collect before the morning standup.

When latency rules it out#

The disqualifier is simple: if a person or a blocking process is waiting on the answer, batch is the wrong tool. Chat interfaces, agents in a tool-calling loop, live RAG answers, autocomplete, and anything inside a synchronous request path cannot tolerate an open-ended wait, even though many batches return in minutes, because you must design for the 24-hour worst case. The window is a ceiling, not a target, and providers offer no intra-batch prioritization. There is also a small operational tax: you need job submission, status polling or webhooks, result parsing keyed on custom IDs, and retry handling for expired or errored items. For a one-off run of 200 requests, that plumbing may cost more engineering time than the discount returns.

If a workload is latency-sensitive but expensive, the levers are different: model routing to send easy requests to a cheaper model, and prompt caching to cut the cost of repeated context. Batch is not a substitute for those; it is the third lever, reserved for the offline slice of your traffic.

Practical takeaways#

Split your LLM traffic into two buckets: "needs an answer now" and "can wait until tomorrow." Everything in the second bucket should default to the Batch API, because 50% off with no quality tradeoff is free money at scale. Tag every request with a meaningful custom ID, design for partial failure, and treat the 24-hour window as the contract even when jobs usually finish in an hour. Stack prompt caching on top whenever requests share a long prefix, and use the separate rate-limit pool to keep big backfills from starving production. Most teams discover that a third or more of their token spend was never latency-sensitive in the first place.