Batch API

A batch API is an asynchronous endpoint that takes a large set of LLM requests in one job, runs them when the provider has spare capacity, and returns the results within 24 hours for half the per-token price. A lot of LLM work fits that contract: embeddings for a million documents, a nightly eval suite, two years of support tickets to classify, yesterday's call transcripts to summarize. None of it needs an answer this second. The same model answers the same prompts, so the discount needs no model change and no prompt change.

Batch API vs a real-time call#

A normal synchronous call promises an answer now. You send one request, the provider reserves capacity to serve it, tokens stream back within seconds, and you pay full price for that reservation. A batch job changes the contract. You bundle requests into one job, tag each with a custom ID so you can match results back, submit it, and walk away. The provider fits the work into idle GPU time between peaks of live traffic and passes the saving back as a discount.

The mechanics are similar across providers. With OpenAI you upload a JSONL file, plain text with one JSON request per line, and create a batch that points at it. With Anthropic you send the requests inline as a list in one call, with no file upload. With Gemini you can send requests inline up to 20 MB or as a JSONL file of up to 2 GB. You then check the job status until it ends. As of October 2026 OpenAI can also notify a URL you register, a webhook, when a batch completes, while Anthropic's batches have to be polled. Results come back in arbitrary order, so the custom ID is the only reliable way to join them to their inputs. Each request in a batch succeeds or fails on its own, so a handful of malformed lines do not sink the other 99,000. A request the provider never reached before the 24-hour expiry comes back marked expired, and Anthropic does not bill for it, so the code that collects results also has to resubmit the expired and failed items by custom ID.

The numbers: 50% off, 24-hour ceiling#

As of October 2026 the three large providers price batch the same way: 50% off both input and output tokens compared with synchronous calls, and results within 24 hours. That holds for Anthropic's Message Batches API, OpenAI's Batch API on the models that support it, which is most of them, and Google's Gemini Batch API. The ceiling is rarely reached. Anthropic says most batches finish within an hour, and OpenAI and Google say many jobs finish well before the deadline without giving a typical time.

The size limits differ. Anthropic accepts up to 100,000 requests or 256 MB per batch, whichever comes first, and keeps results for 29 days. OpenAI caps a batch at 50,000 requests and a 200 MB input file. A worked example shows the size of the saving. Classifying 1 million support tickets with Claude Haiku 4.5 at roughly 600 input and 80 output tokens each is 600M input tokens ($600 at $1 per million) plus 80M output tokens ($400 at $5 per million), so $1,000 synchronous. The same job as a batch costs $500.

Separate limits, and caching on top#

Two other properties can matter as much as the price. Batch jobs draw on their own rate limits. OpenAI counts queued batch tokens against a separate limit from your normal per-model limits, and Anthropic's batch queue limits are separate from its Messages API limits, starting at 200,000 queued requests on the Start tier. You can run a million-item backfill overnight without throttling the production traffic your users depend on, which is hard to do politely through the synchronous API.

The discount also stacks with prompt caching. If every request in the batch shares the same long system prompt or document, cache reads inside the batch bill at the cached rate and are then halved again. On most Claude models a cache read costs a tenth of the base input price, so the combination comes to about 95% off uncached synchronous input. Anthropic treats cache hits inside a batch as best effort, with hit rates it puts between 30% and 98%, so estimate the bill at the low end. For an extraction job where a 5,000-token instruction block repeats across 50,000 documents, putting that block first so it can be cached, before batching, can matter more to the bill than the batch discount itself. Anthropic advises the 1-hour cache duration for batches, because a batch can run longer than the default 5-minute cache lifetime. A 1-hour cache write costs twice the base input price, so against no caching at all it pays for itself once the block has been read twice.

Before either discount, remove the calls you do not need. On our prospect intelligence engine, which runs on Bigcircle's own pipeline, execution is cache-first and deterministic, so a rerun reuses earlier work instead of calling out again. We estimate that cuts repeated external API calls by 50 to 60 percent, and the cost number we watch there is the cache hit rate, not the price per token. We recommend the same order on a batch job: deduplicate the inputs and reuse stored results, then cache the opening block every request shares, then batch what is left, because a request you skip saves its whole cost and a batched one saves half of it.

Flex: the middle option#

Since April 2025 there has been a third choice between a synchronous call and a batch. OpenAI's flex processing, in beta as of October 2026, is a synchronous request billed at batch rates, in exchange for slower answers and the chance of a "resource unavailable" error at busy times, which is not charged. Google's Gemini flex tier, in preview as of October 2026, is half the standard price with a latency target of 1 to 15 minutes, and when it has no capacity it returns an error rather than falling back to the standard tier. Flex suits work that is not urgent but sits inside a pipeline that expects a response to its own request, such as a nightly eval runner that calls the model directly and can retry. It saves you the batch plumbing, and you pay for that with a retry loop.

Where batch fits#

Good batch work has one thing in common: nobody is waiting on any single answer. Evals and regression suites fit, because you rerun thousands of cases against a new prompt or model and only read the aggregate report. So do backfills that apply a new classifier or summarizer to years of records, bulk extraction from contracts, resumes or product listings, embeddings over a large corpus, synthetic data for fine-tuning, and moderation sweeps over archives. Recurring nightly or weekly jobs fit best, because the 24-hour window lines up with a schedule: submit at midnight, collect before the morning standup.

Evals need one split that the usual advice skips. The nightly run of the full suite fits batch. An eval that gates a merge or a deploy does not, because an engineer is waiting on it, and a 24-hour worst case turns a check into a blocked release. We recommend flex or synchronous calls for the gating run, kept to the cases that decide the release, and batch for the full suite overnight.

When latency rules it out#

If a person or a blocking step is waiting on the answer, batch is the wrong tool. Chat, agents in a tool-calling loop, live RAG answers, autocomplete and anything inside a synchronous request path cannot sit on an open-ended wait. Most batches finish well inside the window, but you have to design for the 24-hour worst case, and no provider lets you prioritize one request inside a batch. There is also an operational cost: job submission, status polling or webhooks, parsing results by custom ID, and retrying expired or failed items. For a one-off run of 200 requests, that plumbing can cost more engineering time than the discount returns.

For work that is latency-sensitive and expensive, the levers are different: model routing sends easy requests to a smaller model, and prompt caching cuts the cost of repeated context. Batch sits beside those for the offline share of your traffic. Sorting your calls into "needs an answer now" and "can wait until tomorrow" is the first step, because that split decides which of the three levers each call can use.