A language model becomes an agent through a surrounding loop: receive observations, choose actions, inspect feedback, and continue. Evaluating that loop requires looking beyond a model’s parameter count.
Architecture and training together
The Nemotron 3.5 report introduces Super, a 120B-total/12B-active multimodal model, and Lightning, a 30B-total/3B-active text model. Both use hybrid Mamba–Transformer mixture-of-experts architectures.
The training pipeline combines distillation, long-context training, supervised fine-tuning, and reinforcement learning. Super additionally uses agentic software-engineering mid-training and specialized teacher models. The report also investigates multiple parallel agent attempts followed by self-verification, using the same checkpoint to judge candidates.
Why the harness is part of the experiment
Imagine two evaluations of an identical checkpoint. One agent receives terse tool errors; the other sees structured diagnostics and can inspect a persistent workspace. Even with identical weights, their observations and available recovery strategies differ. A final score therefore describes the checkpoint inside an interaction protocol.
This thought experiment motivates keeping tool definitions, prompts, budgets, and stopping rules with evaluation results. It also suggests a more informative comparison: hold the checkpoint fixed while varying the harness, then hold the harness fixed while varying training.
Reading the evidence
In the cross-harness evaluation, the highest observed harness differs between SWE-bench Pro and Terminal-Bench 2.1. That is evidence that a single preferred harness need not transfer across benchmarks. The paper’s headline Super result is 51.78% on SWE-bench Pro; the separate standard-configuration harness comparison has its own scores and should not be treated as the same setup.
A practical interpretation
Total parameters, active parameters, quality, latency, and total task cost answer different questions. A sparsely activated model may reduce some computation without making environment interaction or long contexts free. Likewise, running several candidate agents can improve selection opportunities while increasing total generation work.
These are useful axes for an agent deployment study: capability per attempt, cost per successful task, and the latency a user actually waits. AgentBaton explores collaboration between models; Polar addresses training under an existing harness.