REINFORCE Zero: aligning execution before correcting policy lag
Why matching the learner and rollout engine changes what a simple asynchronous policy-gradient update can achieve.
Read the noteHao Zhang · Research notes
Thinking about
agents & learning.
The ideas behind the papers. Notes on building, training, and understanding capable language agents.
14 notes · 2025–2026 researchExplore the writing ↓
Writing
From research to intuition.
Each note walks through the problem, the mechanism, the evidence, and what to take away. Start with a topic, or read the complete series below.
14 notes
Nemotron 3.5: treating agent capability as a complete system
A look at how architecture, post-training, agent harnesses, and inference efficiency fit together in Super and Lightning.
Read the noteAgentBaton: knowing when to hand the task over
Model collaboration becomes more interesting when the decision happens inside a task, after useful progress has been made.
Read the noteLoGRA: making the gradient smaller without shrinking the policy
Low-rank gradient sketches reduce RL memory, while predicted-KL control limits how far a compressed update moves the policy.
Read the noteSkill2Env: turning shared expertise into training environments
Agent Skills can describe both the work people want done and the criteria that make an execution useful.
Read the noteOSWorld-Pro: seeing where a computer-use agent gets stuck
A final success score is useful, but subgoals and action-level evidence make failures much easier to understand.
Read the noteAgora: giving research agents a shared experimental memory
A Git-backed contribution graph records what was tried, what was verified, and which findings later work builds on.
Read the noteFlashREINFORCE: one rollout per prompt, one update per batch
A critic-free recipe addresses prompt coverage, stale trajectories, and the extra influence of long negative responses.
Read the noteMolt: keeping agentic RL close to PyTorch
A training framework connects ordinary model code, black-box agents, asynchronous learning, and distributed experience storage.
Read the noteProCUA-SFT: grounding synthetic computer-use data in real tasks
Useful demonstrations need feasible starting states, realistic artifacts, and training examples that match the agent’s inference context.
Read the notePolar: training an agent without rewriting its harness
API-level capture and token-faithful reconstruction let an existing agent participate in a scalable RL pipeline.
Read the noteNemotron 3 Nano Omni: making perception fit inside an agent loop
Multimodal reasoning needs more than input coverage: observation cost and long-context efficiency affect every interaction.
Read the noteProRL Agent: treating rollouts as a service
Decoupling environment execution from optimization makes multi-turn agent training easier to scale and to move between trainers.
Read the noteScaling Up RL: why training longer needs an evolving recipe
Prolonged reinforcement learning raises questions about exploration, reference-policy drift, and how to interpret gains across reasoning domains.
Read the noteA way through the series
Follow the training loop.
01
Collect experience
How agents act, how environments run, and how trajectories reach the learner.
ProRL Agent · Polar · Molt02
Learn from it
Stable asynchronous updates, compact gradients, and longer training runs.
FlashREINFORCE · REINFORCE Zero · LoGRA03
Measure what changed
Training tasks, process-level evaluation, and shared evidence for research agents.
Skill2Env · OSWorld-Pro · Agora