[Databricks AI Research] KARL: Redefining Grounded Reasoning with Multi-Task RL

TL;DR

Training an AI to "search" is fundamentally different from training it to "chat." Databricks AI Research has released KARL (Knowledge Agent via Reinforcement Learning), a system that transforms Large Language Models (LLMs) into elite research agents. By combining a new evaluation suite (KARLBench), an autonomous agentic synthesis pipeline, and a robust off-policy RL algorithm (OAPL), KARL manages to outperform titan models like GPT-5.2 and Claude 4.6 on complex, multi-step enterprise search tasks while being significantly cheaper and faster.

The Motivation: Moving Beyond "Vibe-based" Search

Most current "Deep Research" agents rely on the internet and black-box web search. While impressive, these systems often fail in enterprise environments where data is proprietary, unorganized, and messy.

The authors identified a critical gap: Grounded Reasoning. This isn't just about finding a fact; it's about the iterative process of querying, filtering, and synthesizing information across hundreds of documents. Prior work often optimized retrieval (the "What"), but KARL focuses on the agentic behavior (the "How")—how the model decides to stop, pivot, or compress its history to find a "needle in a haystack."

Methodology: The KARL Architecture

KARL’s success rests on three pillars:

1. Agentic Synthesis (Learning from Self-Play)

Instead of relying on human-labeled data, the team built a pipeline where agents explore a corpus to invent difficult questions.

• Stage I: A generator agent explores a corpus via vector search and proposes grounded Q&A pairs.
• Stage II: Multiple "Solver" agents attempt the task. Only questions that are "just right" (neither too easy nor impossible) are kept for training.

2. OAPL: Stable Off-Policy RL

Traditional Reinforcement Learning from Human Feedback (RLHF) or online GRPO can be notoriously unstable for large-scale Mixture-of-Experts (MoE) models. Databricks introduced OAPL (Optimal Advantage-based Policy Optimization).

• Why it works: It uses a large-batch, off-policy approach that is robust to discrepancies between the training engine and the inference engine (like vLLM).
• Integrated Compression: Crucially, KARL is trained to compress its own context. When the history gets too long, the model summarizes it end-to-end, learning which details are vital for the final answer and which can be discarded.

3. Test-Time Compute (TTC)

KARL utilizes Parallel Thinking and Value-Guided Search (VGS). Instead of one shot, KARL can run N parallel research trajectories and then use a "Generative Aggregator" to synthesize the best findings into a single, cohesive report.

Experiments: Dominating the Pareto Frontier

The researchers evaluated KARL across six search regimes, ranging from financial tabular reasoning to exhaustive entity retrieval.

• Efficiency: RL training actually made the model "smarter" about when to stop. While the base model (GLM 4.5 Air) often performed repetitive, exhaustive searches that led nowhere, KARL learned to "Explore then Commit," significantly reducing token overhead.
• Generalization: Even though it was only trained on two types of tasks (Deep Search and Wide Synthesis), it showed massive improvements on "Out-of-Distribution" tasks like FreshStack (technical docs) and PMBench (internal notes).

Figure: RL training dramatically reduces wasteful post-retrieval searches, proving that the model learns to recognize when it has sufficient evidence.

Critical Analysis: Is it Just "Sharpening"?

A common critique of RL is that it doesn't teach "new" skills but merely "sharpens" the model's existing knowledge. KARL challenges this. By analyzing the Max@K curves, the authors found that KARL can solve problems in 2 attempts that the base model couldn't solve in 16. This suggests that the multi-task RL is actually uncovering new emergent reasoning capabilities.

Limitations

Despite its prowess, KARL still struggles with post-retrieval numerical computation. In some cases, the model would find the correct sports statistics but "give up" because it couldn't (or wouldn't) perform the complex arithmetic needed to verify the final constraint. This highlights that "Search" and "Math" might require separate architectural or reward-based incentives.

Conclusion: The Future of Knowledge Agents

KARL proves that we don't need trillion-parameter models to solve the most difficult enterprise research tasks. By focusing on the reinforcement learning of search trajectories and intelligent context management, specialized agents can outperform general-purpose giants. For the future of AI in law, finance, and engineering, the path forward is clear: agentic self-improvement is the key to mastering grounded reasoning.