Why doesn't temperature zero make an LLM agent repeatable, and what does? At Bigcircle a step that must not vary runs in deterministic code, or through a fixed decision layer the application wrote. We pin the model and serving build, replay from a log, and score repeats so a safety case passes only when every run does.
Key facts
Definition
Repeatable, for an LLM agent, means a replay of the recorded inputs takes the same allowed actions and reaches the same checked state.
How Bigcircle does it
- Bigcircle puts a sum, a date, a permission, and a threshold in deterministic code, and puts a judgement in a fixed decision layer that cannot rewrite its own question.
- Bigcircle writes the model id, the serving build, the prompt version, and the tool results into the run log, so a second pass can replay those inputs.
- Bigcircle reports a pass rate across repeat runs for a quality case, and requires every repeat to pass before a safety case ships.
Comparison
Temperature zero selects the top token, while a change in batch size can change which token is on top.
Key number
A temperature-zero run of 1000 completions produced 80 unique outputs, and the same 1000 matched once batch-invariant kernels were enabled.
The first section explains why the top token can still change when the batch changes. The next gives the counts from one temperature-zero run, beside a figure of the fork. After that come the steps that leave the sampler, the record a replay needs, and the rule for how many repeats have to pass.
Why does temperature zero still change the next token?
Greedy sampling takes the highest-scoring token at every step, and that choice stays put only while the scores stay put. Temperature and sampling is the name of that draw, and temperature zero collapses it onto the argmax. Two calls that share the same scores, the logprobs over the vocabulary, then return the same token, and a swap of the top score is enough to send the next step down another path.
The usual account blames additions that finish in a random order, yet the same multiply on the same inputs matches across repeats. Horace He, with Thinking Machines Lab, reports 1000 such repeats agreeing bitwise with the first, in Defeating Nondeterminism in LLM Inference on 10 September 2025. The forward pass matches across repeats. It does so only when the inputs, including the rest of the batch, stay the same.
Batch size is the input that changes between calls, and batch invariance is the property those kernels lack, so holding the prompt still does not hold the numerics still. Floating-point addition depends on order, so a short batch and a long batch can return different scores from the same row, enough to reorder two close tokens. The server's pass is fixed by the inputs it was given, and the other requests are not an input the caller chose.
Continuous batching changes the neighbours between one call and the next, because the scheduler rebuilds the batch as requests finish and new ones join. That rebuild keeps the GPU busy, and it is also why two calls with one prompt can share the step with different neighbours. The caller does not choose those neighbours, and the prompt text cannot name them either.
How LLM inference works is the serving picture around this batch, and the kernel may pick a reduction that fits the shape it was given. Temperature zero does not see that choice. It only receives the scores that come out. A reduction chosen for a larger batch need not match the reduction chosen when the request ran nearly alone.
What did a temperature-zero run produce?
The same write-up then sampled completions at temperature zero on Qwen/Qwen3-235B-A22B-Instruct-2507, running in non-thinking mode. The prompt was "Tell me about Richard Feynman", each completion ran to 1000 tokens, and the sample held 1000 completions. That sample produced 80 unique completions, and the most common of them appeared 78 times.
Every completion matched through the first 102 tokens, and the first split showed up at token 103. All 1000 had produced the sequence "Feynman was born on May 11, 1918, in" before they split. From that shared opening, 992 of the completions continued with the words "Queens, New York", while the other 8 continued with "New York City". Batch-invariant kernels were then turned on for that same prompt. Every one of the 1000 completions matched.
The companion library reports a second check on a different setup, in the batch_invariant_ops README. Before an upstream change wired the kernels into vLLM, 1000 completions of random length 100 produced 18 unique samples. After the upstream change, one sample remained. The length and the prompt differ from the Feynman run, so the two counts are not one experiment.
The write-up also times those kernels on a one-GPU server running Qwen-3-8B. It answered 1000 sequences, each between 90 and 110 output tokens, in 26 seconds on default vLLM. The unoptimized deterministic build took 55 seconds, and an improved attention kernel brought that down to 42 seconds. On the matmul, the same write-up reports losing about 20% against cuBLAS.
Those figures describe the tokens the server returned, and an agent still has to act on them. A tool result, a clock read, or a page fetched during the run can differ even when the completion would have repeated, and the agent will treat that difference as a new fact. The kernels make the tokens match, and they leave open whether the agent took the same allowed actions and reached the same checked state.
Where does one prompt become two completions?
Two requests can carry the same prompt and still leave on different tokens, which is the fork in the figure below. The figure is a framework and an illustrative model, not a trace, and it states no rates. The counts above are the measurement. The drawing shows where the path splits.
The shared start is one prompt at temperature zero, so the sampler takes the top token on both sides. The left column is a smaller batch and the right column is a larger one, and each column keeps its own addition order. That order is the part floating-point addition cares about.
On one side the top token holds, and on the other a different token moves into first place. From that token onward the two runs feed different text into the next step. A later disagreement can then grow from a split that started as a rounding difference. The model is no longer answering the same prefix, even though the user sent the same prompt.
This figure is an illustrative model of how one prompt becomes two completions, and it is not measured data. A row where the top token changes is a picture of the mechanism, not a claim about how often an agent does it. The run above holds the counts, and a step that must not vary cannot stay in the sampler after a token flips.
Which steps have to leave the sampler?
Code owns every step that has one right answer, because a second sample of that step would be a second action on the customer's system once the write has gone through. A sum, a date, a permission, and a threshold do not become more correct by being drawn again, so we keep them in deterministic code, where the same inputs return the same result. We do that because the number is the action. Drawing it again does not check it.
A model asked to add an invoice can return a different total next time, and the ledger holds whichever total that batch allowed. A reviewer comparing the two totals has no faulty prompt to point at, because both calls did what temperature zero asked of the sampler. The fix is to add in code and hand the model the total already computed.
A judgement is a different kind of step, and it still should not move because the server was busy or a neighbour joined the batch during the call. On the systems we ship, that fixed decision layer is Jev rather than a sentence in the prompt. Introducing Jev into agentic workflows keeps the question text in the application and puts the model output in named fields.
Jev answers only that fixed question. Application code still carries out the outcome. A check that lives in the same prompt as the work gets rewritten when the completion changes, so the question has to stay in the application. Structured output is how those fields arrive, under a schema the application owns.
A missing field counts as a refusal. The next sentence cannot fill that gap. Why AI agents report unfinished work as done is the separate read beside this check. The code reads the state back, and the model's claim that it finished stays out of that read.
| Step | Where it runs | What a second sample would do |
|---|---|---|
| A sum, a date, a permission, a threshold | Deterministic code | Change a value that has one right answer, and write that value through |
| Whether a return matches the contract | Fixed decision layer, question stored in the app | Let a new completion talk the check into a new answer |
| Wording a reviewer could accept in more than one form | The model, then repeat-run evals | Produce another acceptable wording, which a pass rate can absorb |
The last row of that table is allowed to vary, and the first two rows are not. Setting temperature to zero while the permission check stays in the prompt fixes the draw and leaves the decision movable. The next section covers the record a rerun needs, so the second pass is the same test.
What has to be pinned, and what does the log replay?
Versions go into the run record before anyone treats a second pass as the same test, or the comparison is between two systems that only share a name.
We pin the model id, the serving build, the prompt version, and the version of the decision-layer question, and we write each of those into the report the repeat will be judged against. A swap of any of those is a different system, and a regression you cannot attribute is one you will chase in the wrong file.
How to build LLM evals already asks you to pin temperature and the model versions, and to write them into the report. Temperature zero plus those pins reduces the spread and leaves some of it in place, which is why one green run is weak evidence the case will pass on the next attempt. The pin starts the record, and it does not finish the test, because a later repeat can still disagree.
The log replays the inputs the step saw, and it stores tool results rather than calling the live tool again. Call the live tool only when the case under test is the tool itself. We also store the fields handed to the decision layer, the outcome the code applied, and the code version that applied it.
How to build an audit trail for AI decisions keeps the evidence version that was in force when the run decided. A later edit to a linked document must not rewrite what the run was allowed to use, or the replay answers a different case. The repeat has to see that version.
A log that keeps only the final answer cannot separate a batch change from a prompt edit, and the next debugging session will not know which of those moved. You rerun, the answer moves, and nothing in the record says which input moved with it. Dual-layer production agent ops is where that record gets watched after launch, and the trace has to carry the versions plus the replay inputs.
Guardrails on spend, tools, and permissions stay in code even when the log is complete. A replay that sends a payment again, because the log says the model asked, is a second payment. The replay feeds the recorded tool result back in, and the live side effect stays behind the same check it passed the first time.
How do repeat runs decide what ships?
Repeat runs turn one success into a rate, and they turn a safety case into a bar every run has to clear before anyone treats the agent as ready for that action. We take the same case, replay the recorded inputs against the pinned versions, and count how many times the checked state matches. A single green run can be the batch that happened to agree. We keep the rate beside the versions.
Evals here means that case set, built the way the evals guide describes, rather than a score on a public benchmark. The rubric still comes from failures someone has read, and this page adds the repeat on top of that rubric. A quality case, such as a summary a reviewer would accept in more than one wording, ships on the pass rate.
We do not print a universal cutoff, because a rate that fits one product would be a guess on the next. The report shows the rate next to the pinned versions, and the person who owns the standard decides whether that rate is one they will ship. That decision belongs to them, because this guide cannot know the cost of a wrong wording in their product.
A safety case is stricter because one of the repeats is a run a customer can hit. A permission check, an irreversible write, or a required refusal has to pass on every repeat. A single failed run fails the suite. Averaging that failure into a pass rate hides it under the runs that happened to succeed.
Testing MCP servers before agents depend on them uses the same split in another place. A contract test with no model covers the part that must hold, and an agent-in-the-loop eval covers the part a model is choosing. A demo that passed once is neither of those layers, because the next batch can take the other branch.
We run the set when the prompt, the model id, the serving build, or the decision-layer question changes, and again on a schedule. A quiet batch on the day you tested is not the batch your users will share, because their traffic arrives while other jobs are already on the GPU. Observability that stores one trace per case shows only the run you kept, so the repeats belong in the report beside it.
Common questions
Does temperature zero make an LLM agent repeatable?#
Temperature zero does not make the agent repeatable, since it only tells the sampler to take the top token. That token can still change when the request shares its step with a different batch of neighbours. A replay is repeatable when the same inputs produce the same allowed actions and the same checked state. That takes code or a fixed decision layer, pinned versions, a replay log, and repeat runs.
Why do two temperature-zero calls still differ?#
The forward pass depends on the batch it ran in, and kernels without batch invariance add in a different order when that batch changes. A small change in those scores can swap the top token, even while the sampler stays on the maximum. Continuous batching then changes who shares the step as load changes, so two identical prompts need not have been the same computation.
What should stay out of the sampler?#
Sums, dates, permissions, and thresholds stay in deterministic code, because a second sample would write a different value where only one answer is right. A judgement stays in a fixed decision layer whose question text the application owns, with the model output kept in named fields. Wording a reviewer could accept either way can stay with the model, scored as a pass rate across repeats.
What do version pins and a replay log record?#
The pin records the model id, the serving build, the prompt version, and the decision-layer question version, so a second pass names the same system. The log stores the tool results, the decision-layer fields, the outcome code applied, and the code version. A record of only the final answer cannot separate a batch change from an edit someone made on the prompt.
How should repeat runs be scored?#
Run the same case more than once, against the pinned versions and the recorded inputs, and count the passes. For a quality case, report that pass rate and let the owner of the standard decide whether the rate is one they will ship. For a safety case such as a permission or a required refusal, every repeat has to pass or the suite fails.
Can batch-invariant kernels replace the rest of this?#
Batch-invariant kernels can make the completions match, as in the temperature-zero sample that became one identical set of 1000. An agent still calls tools, reads clocks, and fetches pages, so those inputs can differ after the tokens already match. The kernels belong in the serving build you pin, and they do not replace the code, the decision layer, the log, or the repeat-run score.
Further reading
- How to build LLM evals. The rubric, the pin on temperature and model versions, and the per-case results a repeat run needs.
- How LLM inference works. Prefill, decode, the cache, and the batch the request shares.
- Introducing Jev into agentic workflows. The fixed question, the named fields, and the code that still applies the outcome.
- How to build an audit trail for AI decisions. The evidence version that was in force when the run decided.
- Why AI agents report unfinished work as done. The separate read, rather than the model's claim that it finished.
- Dual-layer production agent ops. The trace that has to carry versions and replay inputs.
- Testing MCP servers before agents depend on them. A contract test with no model, beside an agent-in-the-loop eval.
- Temperature and sampling, continuous batching, logprobs, and evals.
- Horace He and Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, 10 September 2025. The 1000 temperature-zero completions, the 80 unique outputs, the split at token 103, the identical 1000 with batch-invariant kernels, the 26, 55, and 42 second server times, and the matmul loss of about 20% against cuBLAS.
- thinking-machines-lab/batch_invariant_ops, README. 1000 random length-100 completions, 18 unique samples before the upstream change, and one unique sample after it.