Can Conformal Self-Verification Improve Web Agent Training and Test-Time Scaling?

CLIFT certifies which self-verification questions to trust

Web agents can already run realistic browser tasks, but reinforcement learning still leans on weak supervision. A binary success signal is too sparse for credit assignment: when every rollout in a GRPO group gets the same outcome, the advantage estimate collapses. A frontier language-model judge gives richer feedback, but it is expensive to call at every step and cannot be assumed available at deployment. CLIFT's answer is to convert judge feedback into a reusable verifier the agent carries with it, rather than a separate process reward model or a post-hoc best-of-N scorer.

The reusable object is a bank of natural-language verification questions about agent behaviour, each carrying a URL scope and a polarity sign. A Compositional Conformal Certifier keeps only questions whose URL-conditional evidence agrees with the training-time judge, using global, host, and URL-path Mondrian-ACI trackers with target alpha-star 0.10, minimum certification count 50, and minimum lift 0.02. Certified questions receive signed trust weights through polarity-aware lift. The bank is not a prompt list: it is a calibrated set of questions, URL scopes, polarity signs, and trust weights that survives training.

URL-local rewards beat one scalar reward per trajectory

Vanilla GRPO broadcasts a single trajectory-scalar reward. CLIFT instead scores each visited URL with verification questions and inherits that URL score to the actions taken there, blending certified verifier evidence into per-step rewards through an asymmetric blend that can support the judge reward but never subtract from it. URL-stratified returns are then written to token advantages over tool-call spans.

The WebArena Infinity ablation isolates this change. Base Gemma-4 with no RL and a greedy rollout scores 61.8% overall. Vanilla GRPO with a scalar comparative-judge reward reaches 64.7%, a 2.9-point gain, while raising average steps on successful trajectories from 15.4 to 18.5. CLIFT's certified step reward plus URL advantages reaches 72.6%, a 10.8-point gain, at 16.0 average steps. The paper reads this as evidence that certified URL-local feedback mainly improves credit assignment rather than encouraging longer trajectories.

Conformal Trajectory Selection adds gains without calling a judge

At test time the certified bank is frozen and reused as structured evidence. The agent samples a greedy rollout plus one or more diverse retries, the self-verifier summarises each URL trace, and a conservative majority-vote rule decides whether to swap away from the current incumbent. No external judge is called. Reported CTS settings use vote count m=3, self-verifier temperature 0.7, greedy incumbent temperature 0, and diverse retry temperature 1.0.

On WebArena Infinity, adding CTS to the trained CLIFT policy raises the overall 9-app success rate from 72.6% to 74.6%, a further 2.0 points, while lowering average steps from 16.0 to 15.5. CTS improves five apps, ties four, and regresses none, with a 17.8% average swap rate. The paper's division of labour is explicit: training produces the stronger policy, and CTS intervenes selectively only when the certified bank supplies enough evidence to prefer an alternative rollout.

A bank trained on an open model still helps GPT-5.5

VisualWebArena is used as a controlled transfer benchmark: three Docker environments with deterministic URL/DOM outcome verifiers, but visual grounding, layouts, and navigation routines that differ from WebArena Infinity's SaaS apps. The question is whether a bank learned while training an open Gemma-4 agent still provides useful evidence when the evaluated policy is GPT-5.5, executor weights are frozen, and only the certified bank is reused at inference.

The Gemma rows isolate the training effect on an open backbone: CLIFT improves the base model, and CTS adds a smaller additional lift. The GPT-5.5 rows test transfer directly. Adding CTS with rollout budget K=4 raises the weighted average to 53.7, narrowly above the WALT reference of 52.9, with gains on all three environments. The paper is careful about what this shows: the human row remains far above every agent, so VisualWebArena is not solved. The claim is narrower and more useful: a bank built while training an open model remains useful when the executor is a closed model.

The certified question taxonomy transfers across benchmarks

Online Mind2Web is the hardest setting: 300 live-web tasks across roughly 30 public domains, judged end-to-end by an LLM rubric rather than DOM matching. No reported policy is fine-tuned on Online Mind2Web data. The only adaptation happens at the verifier layer: questions mined from the WebArena Infinity and VisualWebArena banks are filtered, rewritten, expanded for live-web failure modes such as bot-block pages, hallucinated extraction, partial-filter application, and wrong-final-answer stops, then recalibrated on Online Mind2Web rollouts using the same lift-based certifier.

The transferred taxonomy does not collapse under the noisier live-web judge. Of 132 questions in the final bank, 100 certify, including 75 with positive lift and 25 with negative lift, so the selector receives both progress evidence and failure-mode evidence. The paper treats this as the strongest evidence that the bank is a reusable artifact rather than a benchmark-local prompt list. What transfers is the certified question taxonomy plus the calibration recipe, not the policy: recalibration on a new domain needs only a sample of judge-scored rollouts, with no policy update required to begin selecting trajectories.

Test-time scaling shows up where training leaves room

The Online Mind2Web transfer gains make the selector's contribution visible. On the open Gemma-4 backbone, the transferred bank and CTS raise success from 40.0% to 45.0%, enough to exceed the GPT-5.5 low-effort baseline of 43.7% on the same judge. On the stronger GPT-5.5 high-effort backbone, the same test-time mechanism raises success from 49.7% to 61.0% at rollout budget K=4, a gain of 11.3 percentage points. Every '+ ours' row emits one selected trajectory per task, not an oracle union over samples.

The contrast with WebArena Infinity is the paper's own explanation: there, training already moves the greedy policy close to its ceiling, so CTS adds only 2.0 points. On Online Mind2Web, substantial room remains for the selector to choose among distinct completed trajectories, so the same mechanism contributes far more. The magnitude of test-time scaling therefore depends on how much headroom the trained policy leaves, not on the selector alone.

Uncertified questions are a diagnostic, not a silent failure

CLIFT's failures are designed to be visible. On WebArena Infinity and VisualWebArena, many URL states accumulate enough observations for signed trust weights; on Online Mind2Web, the noisier live-web judge leaves more questions uncertified. The paper frames this refusal as a diagnostic: it signals that the current judge, state partition, or question bank does not support reliable distillation, and it keeps noisy verifier evidence out of both the training reward and the CTS selector. Questions that fall below the lift threshold at all tiers, or whose polarity is contradicted by observed lift, are demoted to neutral or dropped, and can be re-certified later on a different rollout sample.

The study remains bounded in ways that qualify its results. It trains one open-weight backbone and relies on a frontier judge during training and bank construction; if that judge is systematically biased, calibration can detect disagreement only through observed data. The transfer results show the bank can help stronger closed-source agents, but not that one bank is universal across all web domains. The authors also note that stronger open web agents carry misuse risk, so deployment should retain rate limits and domain-specific safeguards.

What to watch when reusing a certified bank

The decision-relevant comparison in this paper is not CLIFT versus a single baseline but the split between training gains and test-time gains, and how that split depends on the benchmark. On WebArena Infinity, training accounts for most of the improvement: 61.8% base to 72.6% with certified step rewards, then 74.6% with CTS. On Online Mind2Web, where no policy is trained, the transferred bank and CTS carry the entire gain, from 49.7% to 61.0% at K=4 on GPT-5.5 high-effort.

For a reader deciding whether to reuse a certified bank, the paper's own variables point to the checks that matter: how many questions survive certification at the relevant URL tier, whether the bank retains both positive-lift and negative-lift questions, and how much headroom the trained policy leaves for selection. The Online Mind2Web bank's 100 certified questions, split 75 positive and 25 negative, and the 17.8% WebArena Infinity swap rate with zero per-app regressions are the concrete operating points to compare against. The most useful reading is that certification quality and policy headroom, not the selector alone, determine whether conformal self-verification pays off.

Sources used in this answer