Why temperature 0 does not give the same answer twice

Setting temperature to 0 tells a language model to take the highest-scoring token at every step, so two runs can only differ if the scores themselves differ. On a shared inference server, they do. Your request is computed in a batch with whatever else is in flight, and the kernels split their work differently for different batch sizes. Floating-point addition gives slightly different results in a different order, so the scores shift in their last bits, and when two candidate tokens are nearly tied, the top one changes and the text diverges. Thinking Machines Lab, which traced the cause on 10 September 2025, asked Qwen3-235B-A22B-Instruct-2507 to "Tell me about Richard Feynman" 1,000 times at temperature 0 and got 80 different completions.

We measured the mechanism on a laptop. On an Apple M2 Pro on 13 October 2026, multiplying one row of numbers by a weight matrix on the GPU gave bit-identical results alone and in batches of two or four rows, then changed 79% to 88% of the output values once the batch reached eight. Repeating the same multiply 1,000 times at a fixed batch size gave identical bits every time. The variation comes from what a request is batched with, which nobody calling a hosted API can see or control.

That is why the vendors hedge. As of 13 October 2026, OpenAI marks its seed and system_fingerprint parameters as deprecated and promises only best-effort determinism, Anthropic says temperature 0 was never fully deterministic and now rejects it on its newest models, and Google Cloud calls its seed best effort. Identical output is achievable when you run the model yourself, at a cost in speed: runs took 25% to 45% longer in SGLang's benchmark and 62% longer in Thinking Machines Lab's own vLLM test.

Temperature 0 means greedy decoding

At each step a model produces a score, called a logit, for every token in its vocabulary. Temperature divides those scores before they are turned into probabilities and one token is drawn at random. As temperature falls towards 0, all the probability piles onto the top score, so inference engines treat temperature 0 as greedy decoding: take the highest score and skip the draw. Setting top_k to 1, or top_p so low that only the top token survives, makes the same greedy choice and has the same problem. (Our explainer on text watermarks walks through the draw itself.)

Two things follow. A seed cannot help at temperature 0, because a seed fixes the random numbers a sampler draws and greedy decoding draws none. And exact ties are not the explanation, despite being a common forum answer: two logits being exactly equal in floating point is rare, and an exact tie would be broken the same way on every run. Some answers also claim that providers quietly replace 0 with a small positive temperature; none of the vendor documentation we read on 13 October 2026 says so, and the variation does not need it. What changes between runs is the scores, each the result of long chains of multiplications and additions whose last bits depend on the order in which the additions happen.

Floating-point addition depends on the order

Computers store real numbers with a fixed number of significant bits, so every addition rounds, and rounding makes the order of addition matter. In float32, (1e8 + 1) - 1e8 is 0, because 1e8 + 1 rounds back to 1e8, while (1e8 - 1e8) + 1 is 1. Neither answer is a bug: each is a correctly rounded sequence of operations.

At scale, the effect is small and constant. We summed the same 1,000 float32 values, ranging from thousandths to thousands, in 10,000 random orders. That produced 37 different totals, from -18163.607 to -18163.533; the sum computed in float64 is -18163.563. A model's forward pass is made of billions of sums like this one, so any change to the order in which a kernel adds things up changes some of its outputs in the last bits.

Repeating the same computation gives the same bits

The usual explanation goes further: GPU threads finish in an unpredictable order, so additions happen in an unpredictable order, so results are random. Some GPU operations do work like that, using atomic additions, and PyTorch offers a switch to avoid them. But Thinking Machines Lab (TML) found that an LLM's forward pass does not rely on them, concluding that concurrency and atomic adds "end up being completely uninvolved". Our run agrees. The same 64-row matrix multiply, repeated 1,000 times on the M2 Pro's CPU and again on its GPU, produced one distinct result on each. For that shape, given identical inputs and an identical batch, the kernels we tested were deterministic.

Determinism of that kind is narrower than it sounds. PyTorch's reproducibility notes state that "Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms." And the inputs to a kernel include more than your prompt: on a server, they include the other rows in the batch.

The batch around your request changes the bits

A matrix multiply computes each output value as a long sum of products. A GPU kernel usually divides the work by giving each core its own tile of the output, which leaves every sum in one piece and its order unchanged. When the batch is too small to give every core a tile, the kernel can instead split each long sum into pieces, compute the pieces on different cores and add the partial sums at the end, a technique called split-K. TML puts it plainly: with a small batch "we may not have enough parallelism and require a split-k matmul". Whether and how a kernel splits the sum, and which kernel configuration runs at all, depends on the shapes involved, including how many rows are in the batch. Change the split and you change the order of addition, so the same row can come out with different bits depending on how many other rows share the multiply. A kernel whose result for one row does not depend on the rest of the batch is called batch-invariant, and most fast kernels are not.

We measured this on 13 October 2026 on an Apple M2 Pro with a development build of PyTorch 2.7.0 and macOS 27.0.1. We multiplied one input row by a weight matrix alone, then as the first row of batches of two to 512 rows, and counted how many output values differed in any bit. The matrices had the shapes of Qwen2.5-0.5B's projection layers (896 by 896, 896 by 4,864 and 4,864 by 896) plus one of 4,096 by 4,096, with random values.

Measured on 13 October 2026 with a development build of PyTorch 2.7.0 (dated 27 February 2025) on the CPU and the Metal (MPS) backend. One row of random inputs against random weights; a value counts as changed if any bit differs.

On the GPU in float32, batches of one, two and four rows matched the row computed alone exactly. At eight rows, 79% to 88% of the output values changed, and from 16 rows up, 95% to 98%. In a batch of eight, putting the row last changed exactly as many values as putting it first, so in this test the size of the batch mattered, not where the row sat in it. The changes are tiny, at most 1.1 millionths at eight rows and 8.3 millionths at larger batches, on outputs of around 1 in size. They are still changes, and nothing in the request reveals them.

The CPU behaved differently. Its float32 and bfloat16 multiplies were batch-invariant at every size we tried, while its float16 multiply changed between two and 17 values as soon as a second row was added. TML makes the same point about servers: the effect is "not unique to GPUs", and endpoints served from CPUs or TPUs have it too. Whether a kernel is batch-invariant is a property of how it is written for each shape and precision, not of the type of chip.

Matrix multiplies are not the only kernels involved. Normalisation layers and attention also reduce long sums. Our plain-PyTorch RMSNorm, which reduces each row on its own, was batch-invariant on both devices, but the fused versions used by serving engines can split long reductions, and TML's fix, as the SGLang team describes it, replaced RMSNorm along with matrix multiplication and attention.

Attention splits its sums too. When a server generates text, each step computes attention for only one new token per request, which is too little work to fill a GPU unless the batch is large. Kernels then split the keys and values into chunks processed in parallel, a technique called split-KV or FlashDecoding, and the usual approach chooses the number of chunks from how much parallelism is needed, so the split, and with it the order of addition, changes with the batch. TML's fix is a fixed chunk size rather than a fixed number of chunks.

Attention also adds a second way for the arithmetic to change: the same token can be computed by different code paths. When a server reads a prompt (prefill), it computes attention for all the prompt's tokens in one pass; when it generates (decode), it computes one new token against stored keys and values. We computed the output for the last of L tokens both ways with PyTorch's scaled_dot_product_attention, using Qwen2.5-0.5B's 14 attention heads of size 64. For sequences of 16, 100, 500 and 2,000 tokens, between 612 and 871 of the 896 output values differed, on the CPU and the GPU alike, by at most 0.36 millionths. Two common server features change how a request is sliced up, and with it how each token's attention sum is divided: chunked prefill, which splits a long prompt into pieces whose boundaries, SGLang notes, can depend on the other requests being processed; and prefix caching, which reuses stored keys and values for a prompt prefix the server has already seen, as described in our explainer on prompt caching. TML found that a kernel which sums over the cached keys and values separately from the new ones gives a token a different order of addition depending on how much of its sequence is already cached.

Our test stops at the kernels. It did not run a language model, so it does not show how often these differences change a token; TML's Feynman experiment, described below, shows that they do.

Run the batch test yourself

This script reproduces the 896-by-896 case. On our M2 Pro it printed 0, 0, 0, 710, 852, 852 and 852 differing values for the seven batch sizes. On other hardware the numbers, and the batch size where they start, will differ, and on some backends they stay at 0.

import torch

if torch.cuda.is_available():
    dev = "cuda"
elif torch.backends.mps.is_available():
    dev = "mps"
else:
    dev = "cpu"
torch.manual_seed(0)
K, N = 896, 896
W = torch.randn(K, N, device=dev) / K**0.5
X = torch.randn(512, K, device=dev)
alone = X[:1] @ W
for B in [1, 2, 4, 8, 16, 64, 512]:
    batched = (X[:B] @ W)[:1]
    print(f"batch {B}: {int((batched != alone).sum())} of {N} values differ")

Why your request's batch keeps changing

Inference servers use continuous batching: at every step they process all active requests together, so the batch a request is in depends on how many other requests are in flight at that moment, and it changes as other requests start and finish during a single generation. TML's conclusion is that nearly all endpoints are nondeterministic because "the load (and thus batch-size) nondeterministically varies". The user cannot see that load and cannot pin it.

On top of the batch sits deployment drift. Providers update kernels, hardware and serving configuration, and each change can move the last bits of every score. OpenAI's Chat Completions reference describes system_fingerprint as the way to tell "when backend changes have been made that might impact determinism". A changed fingerprint explains drift between days; it does not explain two different answers a second apart.

How a last-bit difference becomes a different answer

A difference of a few millionths only matters if it changes which token has the top score. At most steps it does not, because the top token leads comfortably. Over hundreds of tokens, though, some step will have two candidates within a hair of each other, and if the arithmetic drift is larger than the gap, the choice flips. Every later token is generated from a different context, so a single flip can rewrite the rest of the response. In TML's Feynman run, all 1,000 completions agreed for their first 102 tokens. At token 103, 992 continued with "Queens, New York" and eight with "New York City", and the completions kept splitting from there into 80 distinct texts.

Lower precision changes the shape of the problem without removing it. Many models are served in bfloat16, which keeps eight significant bits: near 1.0, neighbouring bfloat16 values are 0.0078 apart. Small arithmetic differences usually round to the same bfloat16 value, which is why our bfloat16 multiplies changed at most two of their 896 to 4,864 output values. When a difference does cross a rounding boundary, the value moves by a whole step. Jiayi Yuan and colleagues found that under bfloat16 with greedy decoding, DeepSeek-R1-Distill-Qwen-7B showed "up to 9% variation in accuracy and 9,000 tokens difference in response length" when only the GPU count, GPU type and evaluation batch size changed (arXiv 2506.09501, revised 24 October 2025). Long reasoning traces give a flip more places to happen and more text after it to change.

On hosted APIs, variation of this size reaches benchmark scores. Berk Atil and colleagues ran five LLMs configured to be deterministic on eight tasks, 10 times each, and saw "accuracy variations up to 15%" between runs (arXiv 2408.04667, last revised 2 April 2025). They measured the variation rather than tracing its cause. It is one reason our ladder for grading AI claims asks for the number of runs before it calls a score measured.

A popular explanation from 2023, a blog post titled "Non-determinism in GPT-4 is caused by Sparse MoE", blamed mixture-of-experts models, in which each token is routed to a few expert sub-networks and a tiny numerical change can send it to a different one. Routing can amplify a difference, but it is not needed to create one. DeepSeek-R1-Distill-Qwen-7B is a dense model, and in the SGLang team's tests of Qwen3-8B, also dense, the same prompt run at varying batch sizes gave up to four different outputs in 50 trials in the engine's normal mode, and prompts cut from the same long text at different prefix lengths, batched randomly, gave up to 18.

What seed and system_fingerprint promise

Vendor documentation promises less than many developers assume. As read on 13 October 2026:

Provider	Seed parameter	What the documentation says
OpenAI, Chat Completions	seed, marked deprecated and beta	"best effort to sample deterministically"; "Determinism is not guaranteed". system_fingerprint is also marked deprecated.
OpenAI, Responses	None	The Responses reference has no seed.
Anthropic	None	"even with temperature of 0.0, the results will not be fully deterministic"
Google, Gemini API	seed	No determinism claim for seed; recommends keeping Gemini 3 models at temperature 1.0.
Google, Gemini Enterprise Agent Platform (formerly Vertex AI)	seed, preview	"best effort"; "Deterministic output isn't guaranteed"
Mistral	random_seed	"different calls will generate deterministic results"
Together AI	seed	"Determinism is best-effort and may not hold across model or backend updates."
Amazon Bedrock	None	Lower temperature "leads to more deterministic responses"; no guarantee.

Sources: OpenAI's Chat Completions reference, Anthropic's Messages API reference, Google's Gemini API reference and Gemini 3 guide, Google Cloud's parameter guide, Mistral's API reference, Together AI's parameter guide and AWS's inference parameters page.

A seed makes sampling above temperature 0 repeatable only if the scores being sampled from are repeatable, which is why every hedge in the table points at the backend. OpenAI's 2023 cookbook on the seed parameter, now marked as archived, already warned of "a small chance that responses differ" even when the parameters and system_fingerprint match. OpenAI's advanced usage guide still presents seed as the way to get "(mostly) deterministic outputs", while the reference marks it deprecated. Mistral's unhedged wording is the outlier; we did not test it, and given the mechanism above we would test it on your own traffic before relying on it.

The newest models narrow the question further. Anthropic's Messages API reference says models released after Claude Opus 4.6 accept no temperature other than 1.0, OpenAI's changelog says GPT-6 Astra accepts no custom temperature or top_p, and Google recommends keeping Gemini 3 models at 1.0. Anthropic's Opus 5.5 migration guide adds that temperature = 0 "never guaranteed identical outputs on prior models". Our guide to why new model APIs reject old parameters covers those restrictions and what replaces each one.

Getting identical output: batch-invariant inference and its cost

If you run the model yourself, identical output at temperature 0 is achievable by making the kernels batch-invariant. TML released batch-invariant kernels for matrix multiplication, RMSNorm and attention, the last with a fixed split size, so that how each sum is divided no longer depends on the batch. With them, all 1,000 Feynman completions were identical. The price is speed: TML says its batch-invariant matrix multiply gives up about 20% against cuBLAS, and in its vLLM benchmark a run that took 26 seconds by default took 55 seconds with unoptimised deterministic kernels and 42 seconds with an improved attention kernel. TML also shows why this matters beyond tidy logs: when the sampler and the trainer in reinforcement learning compute slightly different numbers, training that is meant to be on-policy quietly is not, and with batch-invariant inference the KL divergence between the two was 0.

Two open serving engines have since built this in:

• vLLM. Setting VLLM_BATCH_INVARIANT=1 turns on batch invariance, in beta, on NVIDIA GPUs of compute capability 8.0 or newer, AMD Instinct GPUs and Intel XPUs. The documentation warns that it "may impact performance" and lists the models it has been tested with.
• SGLang. The --enable-deterministic-inference flag, from version 0.5.3, works with chunked prefill and CUDA graphs. The SGLang team's 22 September 2025 measurements on Qwen3-8B found "most slowdowns ranging from 25% to 45%", with an average of 34.35% across its two fastest attention backends. At the time it supported dense models only.

The LLM-42 paper (Raja Gond and colleagues, January 2026) argues that batch-invariant kernels charge every request for the few that need determinism. In its measurements, one request in 11 asking for determinism under SGLang's deterministic mode made decode throughput "collapse by 56%", from 931 to about 415 tokens a second, because the whole batch has to run on the slower kernels. LLM-42 decodes on the normal fast path instead, then replays candidate tokens with a fixed reduction order, keeping those that match and rolling back those that do not. It reports 911 tokens a second in the same scenario, within 3% of the non-deterministic throughput. It is a research system, not a feature of any hosted API.

More precision is a different trade. Yuan and colleagues propose LayerCast, which stores weights in 16 bits but computes in float32, making rounding differences smaller and flips rarer. It narrows the problem rather than closing it: our float32 multiplies still changed most of their output bits with the batch size.

We found no hosted API that documents a batch-invariant or deterministic mode as of 13 October 2026, among OpenAI, Anthropic, Google, Mistral, Together AI, Fireworks, Baseten, OpenRouter and Amazon Bedrock. That is an absence of documentation, not proof that none exists.

What to do instead of comparing strings

• On a hosted API, assume the same prompt can return different text, whatever you set. Test what matters about an output, such as its structure, the facts in it, whether its code runs or whether a parser accepts it, and use structured outputs where you need a fixed shape. A fixed shape is not a right answer: in our test of six cheap models on JSON extraction, all 3,120 strict-schema responses had the right shape, yet 91 had at least one wrong value.
• When a result must be reproducible later, for an audit trail, a regression baseline or a cached answer, store the response. Regenerating it is not a reproduction.
• If you use seed on OpenAI's Chat Completions, log the seed and system_fingerprint with every response, and treat a difference as a regression only when the fingerprints match. Both parameters are deprecated, and even matching fingerprints leave "a small chance" of different text.
• To see where a response is fragile, request log probabilities where the API offers them. Steps where the top two tokens are close are the ones most likely to flip.
• When you compare models or prompts, run each case several times and report the spread. Differences of a few points from a single run are within what Atil's and Yuan's teams measured from nondeterminism alone. For coding scores, nondeterminism is not the only source of error: our analysis of leaking coding benchmarks covers the routes by which an agent can find a fix instead of writing it.
• If you need bitwise-identical output, for debugging, reinforcement learning or an auditor, self-host with batch-invariant inference. Pin the model weights, engine version, GPU type and count, precision and parallelism, and budget for runs 25% to 62% slower, the range in SGLang's and TML's published tests. Then test it the way the problem works: run the same prompt alone and inside a full batch, and compare bytes.

The decision rule: if an output must be identical, store it or generate it on infrastructure you have made deterministic; if it only needs to be correct, test it for correctness and stop expecting the same string twice.