RL training must store more than model weights. Gradients, optimizer state, activations, and rollout payloads all compete for memory. A model that can run inference may still be expensive to update.
Compress the learning signal
LoGRA uses low-rank gradient sketches for RL post-training. For a weight matrix with gradient G, it forms a compact projection S = GAᵀ, where A is a random projection held fixed during gradient accumulation. The implementation accumulates projected contributions directly instead of first materializing a full accumulated gradient.
The compact factors support weight updates and policy synchronization. Predicted-KL control estimates the policy change caused by a proposed update and rescales its magnitude to fit a budget. The method describes both the sketch and step-size control.
A memory calculation
As an illustrative shape calculation, a 4,096 × 4,096 gradient contains about 16.8 million values. A 4,096 × 256 sketch contains about 1.05 million. This comparison concerns just those arrays; it is not an estimate of total training-memory savings.
That distinction matters because model weights and activations do not disappear when a gradient buffer becomes smaller. A complete memory profile must say which allocations changed and which stayed constant.
The evidence and its boundary
The report finds up to 45.7% average training-memory reduction in its evaluated reasoning settings. It also demonstrates more than 1,100 training steps for a 27B model on one eight-GPU node, where dense Adam does not fit. The latter is a feasibility result; it cannot supply a matched dense-Adam accuracy comparison at that model size.
Why update size deserves its own control
Parameter distance and output-distribution distance are different quantities. Even if a compressed direction is useful, multiplying it by a large step can change action probabilities too abruptly. Conceptually, direction selection asks “which way?” while a trust budget asks “how far?” They should be inspected independently.
Memory savings are most useful when learning progress and synchronization costs remain acceptable. Molt considers the wider memory and execution pipeline, while Scaling Up RL examines stability over longer runs.