Agentic Verifier: Redefining Reward Modeling via Multi-Turn Deliberation and Tool-Augmented RL
AgentV-RL: Scaling Reward Modeling with Agentic Verifier
The paper introduces Agentic Verifier, a multi-turn, tool-augmented reward modeling framework that treats solution verification as a deliberative process. By coordinating bidirectional (forward and backward) agents and specialized RL training (AgentV-RL), it establishes a new SOTA on reasoning benchmarks, with its 4B variant notably outperforming 70B outcome-level reward models.
TL;DR
Researchers from Fudan University and ByteDance have introduced Agentic Verifier, a paradigm shift in how we judge LLM outputs. Instead of a single-turn scalar reward, this framework uses bidirectional "Agentic" reasoning and Python tool use to scrutinize solutions. Their 4B model, trained via a recipe called AgentV-RL, now outperforms 70B models, providing a more reliable "compass" for Test-Time Scaling (TTS).
The "Superficial Plausibility" Trap
The scaling of inference-time compute (e.g., Best-of-N or sequential refinement) relies entirely on a Verifier to pick the right path. However, current Outcome Reward Models (ORMs) and Process Reward Models (PRMs) suffer from two fatal flaws:
- Error Propagation: LLMs are often "tricked" by incorrect solutions that look like legitimate reasoning.
- Lack of Grounding: Verifiers frequently hallucinate during complex calculations or knowledge-heavy steps because they operate purely in bitspace without symbolic tools.
In the figure above, note how standard GenRM is misled by a flawed solution, whereas the Agentic Verifier leverages external grounding to catch the logic gap.
Methodology: The "Plan-Validate-Verdict" Pipeline
The authors propose a bidirectional multi-agent framework:
- Forward Agent (Sufficiency): Traces the logic from premises to conclusions.
- Backward Agent (Necessity): Reasons in reverse, ensuring the final answer is actually supported by the problem constraints.
Both agents follow a structured Plan-Validate-Verdict strategy, where they can invoke a Python interpreter to verify numerical steps.
AgentV-RL: Distilling Agency
To make this efficient, the team introduced AgentV-RL. This isn't just a training set; it’s a scalable recipe:
- Synthetic Trajectory Sampling: Automatically generating verification logs (including tool calls).
- Rejection Fine-Tuning (SFT): Teaching the model the "behavior" of a systematic verifier.
- Reinforcement Learning (GRPO): Using Group Relative Policy Optimization to optimize the model's exploration of verification paths without needing a complex value model.
Experimental Breakthroughs
The results prove that "thinking about judging" is as important as "thinking about solving."
- SOTA Achievement: The Agentic-Verifier-Qwen3-4B reached 79.0% on MATH500. For context, the 70B Llama-based ORM only reached 54.6%. That is a +25% jump with a model 1/17th the size.
- Robust Scaling: Unlike many verifiers that plateau, the Agentic Verifier shows consistent gains as you scale Best-of-N samples from 32 to 128.
- Generalization: The model successfully generalized to coding (LiveCodeBench) and multi-hop QA (HotpotQA), demonstrating that the "agentic" approach isn't limited to math.
Critical Insights: Why it Works
The success of the Agentic Verifier stems from deliberation. By forcing the model to create a "plan" and "verify sub-steps" before giving a verdict, the authors introduce a structural bottleneck that prevents the model from jumping to conclusions. The Backward Agent is particularly innovative—it acts as a "sanity check" that catches errors the standard forward-pass often ignores.
Limitations
The primary trade-off is Latency. As shown in the paper's analysis, the Agentic Verifier requires significantly more tokens and rounds (11.3 rounds on average) compared to base models. In real-world production, this suggests a need for tiered verification systems where simple problems use fast ORMs and complex ones trigger the Agentic Verifier.
Conclusion
Agentic Verifier proves that Reward Modeling is no longer just a classification task; it is a reasoning task. By treating the verifier as an autonomous agent with the power to use tools and double-check its own logic, we unlock the true potential of Test-Time Scaling.