Deterministic Inference for LLMs

Abstract

Large language models (LLMs) exhibit surprising non-determinism: identical prompts, under supposedly deterministic configurations (temperature=0, greedy decoding), can still produce divergent outputs. This undermines reproducibility, a cornerstone of both science and engineering. While the common explanation points to floating-point rounding and GPU concurrency, we argue that nondeterminism primarily arises from batch-size-dependent kernel numerics, which change the effective sampling distribution of model outputs. By reframing the issue statistically, we show how nondeterministic inference can be understood as uncontrolled variance injected into the system. We then describe engineering methods to achieve batch-invariant kernels and quantify the trade-offs. Finally, we argue that deterministic inference enables more rigorous evaluation and enables truly on-policy reinforcement learning.

Keywords: Large language models, nondeterminism, deterministic inference, batch size, kernel numerics, reproducibility

In statistics, reproducibility can be compared to controlling variance. When a system outputs different results from the same inputs, we can think of its variance as non-zero even under fixed conditions. For LLM inference, the "noise" is not intentional randomness (like temperature > 0) but uncontrolled variance introduced by hardware implementation details.

This implies that instead of sampling from a fixed distribution P(y|x), the system effectively samples from a family of slightly different distributions depending on batch size, kernel tiling, or server load. From the user’s perspective, the model’s conditional distribution is ill-defined. This is equivalent to adding hidden nuisance parameters into the sampling process.

Floating-Point Arithmetic: Bias and Variance

Floating-point non-associativity is the seed of divergence. The IEEE 754 standard defines floating-point addition as non-associative, meaning (a + b) + c ≠ a + (b + c) due to rounding errors. The operation order can alter rounding, producing multiple possible outcomes. Two clarifications help frame this statistically:

Bias: The effect introduces bias relative to exact arithmetic (since finite-precision results systematically deviate).

Variance: However, when the order of operations changes nondeterministically across runs, it adds variance on top of that bias.

Thus, nondeterminism is not just about inaccuracy, but about run-to-run variance. Crucially, in many GPU kernels, the operation order is actually fixed, making variance zero for repeated runs under identical kernel configurations. So where does the variance re-enter?

Batch Invariance: Controlling Nuisance Variables

Inference servers batch requests dynamically. Suppose kernel output K(X, W, B) depends not only on the input X and model parameters W, but also on the batch configuration B (number of requests, tiling strategy). If B varies unpredictably with server load, then the marginal distribution P(y|x) = ∑ P(y|x, b)P(b) is effectively a mixture distribution over many batch-dependent sub-distributions. This mixture is what users experience as nondeterminism.

Batch size B behaves like an unobserved confounder: its random fluctuations induce spurious variance in outputs. Achieving batch invariance is equivalent to conditioning away this nuisance variable, enforcing P(y | x, b1) = P(y | x, b2).

Case Studies: Kernels as Sources of Variance

RMS Normalization

RMSNorm computes a mean-square reduction across features. Parallel implementations typically change reduction strategy (e.g., from thread-local to warp-shuffle reductions) at small vs. large batch sizes. The estimator used to calculate the mean-square changes its form depending on B, yielding different rounding noise profiles. By fixing the reduction strategy, we hold the estimator constant and remove batch-induced variance.

Matrix Multiplication

Matrix multiply is a summation estimator over dot products. Adaptive kernels (using different tile sizes or split-K strategies) correspond to switching between different unbiased estimators with different finite-precision variances. If the estimator choice depends on batch size, then variance is injected stochastically. Fixing a single estimator across shapes stabilizes results, even if not variance-optimal for each shape.

Attention Mechanisms

Attention reduces across both sequence and feature dimensions. Optimizations like chunked prefill or KV-cache splitting are equivalent to changing the decomposition of the estimator for the softmax denominator. This is like switching between stratified sampling schemes, unbiased in expectation but with different variance characteristics. When which scheme is chosen depends on server load, the effective sampling distribution for tokens drifts nondeterministically. Batch-invariant attention kernels eliminate this uncontrolled design switching.

Measuring Nondeterminism: Empirical Variance

To quantify nondeterminism, consider repeated completions of the same prompt at temperature 0. If inference were deterministic, the empirical distribution would have zero variance, every trial identical.

In practice, one study with a Qwen-235B model produced 80 distinct completions across 1000 trials. Divergence occurred at token 103, with completions splitting between "Queens, New York" and "New York City". From a statistical perspective, the system induced an artificial branching in the probability distribution. When batch-invariant kernels were enabled, variance collapsed to zero: all 1000 completions were identical, as theoretically expected.

Performance Tradeoffs: Variance vs. Efficiency

One often trades variance for bias or efficiency. Similarly, deterministic inference trades efficiency for reduced variance. Benchmarks with Qwen-8B showed:

This ~60% slowdown is non-negligible but far from catastrophic. More importantly, it yields a zero-variance system where reproducibility is guaranteed.

Implications for Reinforcement Learning

Deterministic inference is not merely a debugging convenience; it is mathematically essential for Reinforcement Learning from Human Feedback (RLHF), particularly for algorithms like Proximal Policy Optimization (PPO).

RLHF relies on the Kullback-Leibler (KL) divergence to constrain the updating policy πθ so it does not drift excessively from a reference policy πref. The KL divergence DKL measures the amount of information lost when one distribution approximates another, defined formally as:

DKL(πθ || πref) = ∑ πθ(x) log( πθ(x) / πref(x) )

With nondeterministic inference, even temperature=0 rollouts do not correspond exactly to the training distribution. Inference randomness introduces a noise term ε into the log-probabilities log πθ(x). Because the KL term involves the ratio of probabilities, this noise is amplified, effectively increasing the variance of the KL estimator. The optimizer cannot distinguish between "good" policy exploration and "bad" hardware noise.

Experiments have shown that without deterministic correction, the estimated divergence grows artificially, causing the penalty coefficient β to fluctuate wildly and leading to reward collapse. However, with deterministic inference, the hardware-induced variance is removed (Var(ε)=0), ensuring that DKL reflects only true policy changes. This allows for unbiased gradient estimates and enables true on-policy reinforcement learning.

Conclusion

Nondeterminism in LLM inference is not a mysterious inevitability of floating-point math. It is a statistical variance problem caused by kernel implementations that alter reduction strategies with batch size. By enforcing batch invariance, we remove this hidden source of variance, enabling deterministic inference.

While the primary discussion has focused on reproducibility in research, the implications of deterministic inference extend into critical application areas where auditability is mandatory. With careful kernel engineering, it is achievable, transforming LLMs from stochastic black boxes into reliable infrastructure.