Long agent trajectories finish at unpredictable times. If an update needs several sibling rollouts of each prompt, the slowest sibling can hold back otherwise useful experience.
Three coupled decisions
FlashREINFORCE centers rewards across a fresh batch of independent prompts, uses token importance sampling with a sequence-level drift gate, and averages within each trajectory before averaging across the batch. Each batch receives one update and is then discarded. No learned critic is required.
The sequence gate matters because correcting an action probability at a stored history is not the same as correcting the distribution of complete histories. The project’s method explanation makes this distinction explicit.
Why length changes an update
Imagine two failed attempts with the same negative advantage. One contains 100 action tokens and the other 1,000. If their per-token contributions are simply summed, the longer attempt can receive ten times the total weight solely because it is longer.
Averaging within an attempt changes this weighting choice. It does not discover which individual step caused failure, and it does not turn a coarse task reward into perfect credit assignment. It makes a narrower decision about how much influence trajectory length should have.
What the reported comparison says
The paper reports 38.0 mean accuracy across five reasoning benchmarks for Qwen2.5-Math-1.5B using 256K rollouts, versus 36.3 for the reported GRPO comparison using 512K. Another experiment remains stable through 6,000 updates at approximately four updates of policy lag. Those comparisons concern their stated models and training recipes.
Freshness has two meanings
An experience can be new to the optimizer while having been generated by an older policy. Distinguishing unused data from current-policy data helps explain the design: one-pass training controls reuse, while behavior correction and drift screening address policy mismatch.
A useful reproduction would inspect prompt coverage, rejection rates, response lengths, and success together. This is a suggested analysis, not a claim that one statistic alone proves stability. Compare with REINFORCE Zero, which puts numerical alignment at the center of its asynchronous recipe.