A rollout is more than model generation. A coding agent may need a repository, dependencies, a running sandbox, several tool calls, and a test suite before a reward can be assigned.
Separate execution from optimization
ProRL Agent serves the multi-turn rollout lifecycle through an API. It separates the rollout service from the RL trainer and supplies extensible sandbox environments, including support for rootless HPC settings. The system is integrated into NVIDIA NeMo Gym.
The server design distinguishes initialization, agent execution, and evaluation. These stages have different bottlenecks: initialization can be I/O-heavy, execution depends on model inference, and scoring can range from a quick check to a long test suite. The experiments cover software engineering, math, STEM, and coding tasks.
Why stages should overlap
Consider an illustrative worker that spends 20 seconds preparing a sandbox, 60 seconds running the agent, and 20 seconds scoring its output. A strictly sequential worker uses only one stage at a time. Multiple in-flight jobs create an opportunity to prepare one environment while another agent runs and a third result is evaluated.
Those example times are not paper measurements. They show why separating stages can expose unused resources. Actual speedup depends on capacity at every stage; adding concurrency cannot eliminate the slowest shared bottleneck.
A service contract is a research interface
A trainer needs to know what it submitted, how to retrieve a result, and how to handle an interrupted rollout. The environment service needs to know how to execute and score the task. Keeping those responsibilities explicit makes it easier to change an optimizer without reimplementing the environment.
Decoupling also moves integration questions into the contract. A useful implementation must still define which policy produced the trajectory and which tokens are actions. Distributing execution without preserving that information would make the system faster at producing ambiguous training data.
Following the progression
The paper establishes a rollout-service design rather than a universally best RL objective. Polar continues this direction with black-box harness capture and token-faithful reconstruction. Molt connects agent execution to a broader asynchronous training and storage stack.