Asynchronous RL lets agents keep working while a learner updates their policy. That improves resource use, but it complicates a basic question: how probable was the action that produced this experience?
Two disagreements hiding in one ratio
REINFORCE Zero separates weight lag from numerical execution mismatch. Older rollout weights create the first. Different kernels can create the second even when weights and token histories agree. Trainer–inference invariance, or TII, aligns execution; unclipped token-level importance sampling then corrects the remaining weight lag.
The recipe uses recorded behavior probabilities, group-mean advantages, and averaging within each response. It does not require ratio clipping or sample-rejection gates. The algorithm guide describes the update and alignment workflow.
An intuition for the correction
Suppose an action was sampled with probability 0.2 and the current policy assigns it 0.3. Its importance weight is 1.5. This elementary example assumes that 0.2 really describes the sampler. If a later recomputation supplies a different denominator, the update answers a different statistical question.
This is why logging the sampler’s probabilities and checking execution agreement are separate engineering tasks. One preserves the evidence; the other makes the computational paths comparable. A loss function alone cannot repair missing provenance about how a token was sampled.
What the experiments establish
The reported experiments sustain 20,000 reasoning updates at nominal lags 2 and 8. A 16-GPU comparison reports 2.40× training-step throughput versus synchronous TII across 12,000 updates; evaluation time is excluded. The nominal lag-8 run has a measured mean lag around 10.5, so the configured lag is not the entire staleness distribution.
What to carry into a new system
A useful diagnostic order is to compare identical weights and histories first, then introduce asynchronous weight updates. Otherwise, two sources of disagreement change simultaneously and the resulting instability is hard to interpret. This is an experimental-design recommendation, not a guarantee that every model or kernel stack will behave identically.
Read this alongside FlashREINFORCE for a different treatment of stale trajectories, and Molt for the training infrastructure.