One-Eval: Revolutionizing LLM Evaluation via Agentic Workflows
One-Eval: An Agentic System for Automated and Traceable LLM Evaluation
One-Eval is an agentic evaluation system that automates the end-to-end LLM assessment process by converting natural-language requests into executable workflows. It integrates three core components—NL2Bench, BenchResolve, and Metrics & Reporting—to achieve SOTA levels of automation, enabling a 84% full-plan success rate without manual intervention.
TL;DR
Evaluating Large Language Models (LLMs) has become a bottleneck in AI development due to the manual effort required to find benchmarks, configure environments, and interpret scores. One-Eval is a new agentic system that transforms a simple natural language prompt—like "I want to test my model's math and general knowledge"—into a fully executable, traceable, and diagnostic evaluation report. It achieves an impressive 84% zero-shot success rate in generating complete evaluation plans.
The Pain Points: Why Evaluation is Broken
Currently, practitioners face a "fragmentation tax." To evaluate a model, one must:
- Discover: Find the right benchmark (e.g., MMLU, GSM8K).
- Resolve: Deal with broken HuggingFace links or incompatible dataset versions.
- Map: Manually align dataset columns (e.g., "question" vs. "prompt") to the runner.
- Analyze: Stare at a single accuracy score without knowing why the model failed.
Existing tools like lm-eval-harness or OpenCompass solve the execution part but leave the planning and analysis to human experts.
Methodology: The Three Pillars of One-Eval
One-Eval decomposes the evaluation lifecycle into a modular, three-stage agentic pipeline supported by LLM-driven agents.
1. NL2Bench (Intent to Plan)
This module translates vague user requests into a structured EvalPlan. It uses a dual-retrieval strategy (Embedding-based for semantics and TF-IDF for keyword accuracy) to search both a local gallery of 77 curated benchmarks and the vast HuggingFace Hub.
2. BenchResolve (Resolution & Normalization)
Once benchmarks are selected, the BenchResolveAgent handles the "dirty work." It automatically identifies the correct dataset splits (e.g., choosing validation if test is missing) and maps heterogeneous data fields into a unified internal schema.
Figure 1: The One-Eval architecture showing the flow from natural language request to decision-oriented reporting.
3. Metrics & Reporting (Beyond the Score)
Unlike traditional tools, One-Eval doesn't just output a number. It recommends specific metrics (e.g., symbolic_MATCH for math) and generates Diagnostic Reports including radar charts, root-cause analysis of failures (Hallucination vs. Logic error), and case studies.
Experimental Validation
The authors tested One-Eval on 100 diverse requests across domains like coding, safety, and reasoning.
- Executability: 99% of requests generated a valid initial plan.
- Reliability: 84% of requests completed the entire pipeline—from intent to metric selection—without any human intervention.
- Efficiency: The average end-to-end setup time was roughly 13 minutes, a fraction of the time a human engineer would take to manually configure five different benchmarks.
Performance Comparison
One-Eval stands out by providing native support for automated benchmark and metric recommendation, features largely absent in current industry-standard frameworks.
Table 2: One-Eval vs. existing frameworks like OpenCompass and HELM.
Deep Insight: Why It Works
The "secret sauce" of One-Eval is its Hierarchical Resolution Strategy. By maintaining a local registry of "expert-validated" configurations for common benchmarks while allowing a "dynamic fallback" to HuggingFace for long-tail datasets, it balances the stability of a static tool with the flexibility of an LLM agent.
Furthermore, the inclusion of Human-in-the-Loop (HITL) checkpoints acknowledges that evaluation is a subjective science. By allowing users to edit the EvalPlan before execution, the system maintains trust without sacrificing the speed of automation.
Conclusion & Future Outlook
One-Eval represents a significant step toward Autonomous AI Engineering. By treating evaluation as a traceable workflow rather than a script, it provides the "evidence trail" necessary for industrial auditing and debugging. Future iterations looking to expand into multimodal tasks (video/audio) will likely set the standard for how we certify the next generation of Foundation Models.
Project Link: GitHub/OpenDCAI/One-Eval