Can recursive self-improvement improve 3D medical image perception agents?

Recursive improvement needs to know which stage failed

A missed lesion in a three-dimensional scan can fail at several different points. The relevant view may never have been rendered, the reader may have seen it but not recognized it, detection may be inconsistent across reads, or a correct candidate may be discarded during aggregation. Each failure calls for a different fix, and improving the wrong component leaves the real bottleneck in place.

The paper argues that logged agent trajectories cannot resolve this. A trajectory shows what the agent inspected, not whether an unvisited slice or crop would have exposed the finding, and self-generated judgments do not independently establish whether a predicted location was correct. ReVision3D therefore separates two jobs: attribution, which traces a missed lesion to the stage where evidence was lost, and external verification, which tests whether a proposed revision actually improves visual capability.

Annotated volumes make counterfactual views checkable

The central mechanism is what the authors call an exact replay world. Given a scan and its reference masks, a view-selection program can be scored on the lesions it would geometrically expose without running the reader at all. A lesion counts as exposed when a view plane intersects its extent or passes within a fixed spatial tolerance of 5 mm. Exposure and detection are kept distinct: a lesion can be covered by a rendered view and still be missed.

The same tolerance supports grounded verification. Each predicted location is matched to the nearest reference lesion within 5 mm, producing distinct hits, duplicates, and ungrounded findings. These quantities feed a training reward with bounded weights, while reference masks are used only by replay and verification and are never shown to the reader. Promotion to the next round is decided by a separate fixed objective J, which macro-averages recall over false-positive budgets of 0.25, 0.5, 1, and 2 per patient across organs.

Four intervention levels map to four failure modes

The failure decomposition is an exact, non-negative identity over missed lesions: not shown, shown but never detected, detected only when sampled, and detected then discarded. Each term routes to one editable level. Not shown points to acquisition, where a program requests at most a bounded number of additional views and candidates are screened geometrically. Shown but not detected points to perception, where the adapted vision modules, LoRA rank, and input resolution are changed and candidates require actual training.

Inconsistent detection points to the training recipe, where group-relative optimization learns only from prompts with non-identical sampled rewards. Discarded findings point to inference, where a score merges 3D candidates using label-free features such as cross-slice support, and cached outputs allow candidate programs to be evaluated directly on J without retraining. The designer, a frozen language model, proposes revisions only within the level selected for that round, and the verifier and system objective stay fixed so the designer cannot redefine success.

A learnability census splits the undetected loss

The never-detected term mixes two situations that call for different responses: lesions the current reader can catch on some sampled reads, which may still respond to training, and lesions never caught at all, which more strongly suggests a perception limit. The paper estimates this split with a learnability census that stores sampled rewards and computes the fraction of lesion-bearing prompts sitting on the learning frontier.

A larger learning-frontier share assigns more of the undetected loss to the training recipe; a larger never-hit share directs the search toward perception. The authors are explicit that this split is a routing heuristic rather than an exact decomposition, because it relies on finite sampled reads. Each accepted revision triggers renewed attribution, so the dominant bottleneck can shift from round to round, and an evidence-backed playbook carries successful and failed hypotheses forward.

Only gains beyond seed noise are kept

Acceptance is deliberately conservative. Training revisions are evaluated on matched development patients and seeds and accepted only if the change in J exceeds the incumbent's seed range and the minimum per-seed change is positive. Inference revisions require at least 0.9 probability of a positive change under a paired bootstrap. Screens rank candidates but never promote them.

This matters for reading the headline numbers. The reported 79% liver recall and 83% kidney recall at under 0.4 false positives per patient come after perception was identified as the dominant remaining limitation and revised. Recall is pooled over lesions of at least 1 mL, false positives are counted per patient, and uncertainty is a patient-level bootstrap with 2,000 resamples. These are benchmark results on public CT data, not a clinical claim.

Better views alone barely help frozen readers

A separate component experiment isolates acquisition. Thirteen frozen frontier readers and the paper's own frozen-encoder reader saw the same 31 held-out liver patients under three view conditions: the standard eight-slice survey, views chosen by the acquisition program, and oracle slices passing through every annotated lesion. On the survey, frontier-model recall ranged from 0.05 to 0.36, with the best reader at 0.36 recall and 0.19 false positives per patient.

Changing only which slices were shown produced only modest gains, and the paper reports that better views do not help frontier readers. That comparison is what makes the perception result interpretable: if exposure were the binding constraint, oracle slices would have closed much of the gap. The acquisition study is also reported as a separate component experiment under the budgeted-read protocol with geometric screening only, because it was not submitted to the full promotion gate.

Small lesions gain most, and the loop stays bounded

The largest gains are reported on small lesions, which is consistent with the paper's own observation that a four-slice survey exposes only 83% of liver lesions and that a slice within 5 mm of a small lesion may show only its edge. This is why the study also reports lesion-guided slices and a dense protocol showing every axial slice at 5 mm, roughly 43 prompts per patient, with findings merged within 12 mm in 3D.

Two boundaries shape how far the result travels. The intervention space is bounded to four predefined levels, so the loop cannot introduce a new backbone, modality, rendering operator, or reasoning strategy unless that capability is exposed as an action. And replay is exact for geometry but not for perception: acquisition and inference can be screened cheaply, while perception revisions still require training and evaluation. The authors also note that better lesion localization may not translate into better downstream clinical decisions, since the fixed objective measures spatially verifiable perception rather than clinical utility.

Attribution, not more compute, is the deciding variable

The paper's most useful comparison is not between model sizes but between where improvement effort is spent. Frozen frontier readers given oracle lesion slices still stayed far below the revised reader's recall, while the accepted perception revision reached 79% liver and 83% kidney recall at under 0.4 false positives per patient. In this setup, identifying the limiting stage and verifying changes against fixed geometry did more work than adding visual exposure or a stronger base reader.

The practical reading is conditional: recursive self-improvement of a 3D medical perception agent is worth pursuing when annotated volumes can serve as a replay world, when the verifier and objective are held fixed, and when acceptance is gated on gains beyond seed noise. Where those conditions hold, the priority is to attribute the loss before revising, and to expect the dominant bottleneck to move between rounds rather than stay at perception.

Sources used in this answer