A decision made before the customer asks
The study frames proactive retail assistance as a single decision point. From sparse third-person video recorded before any explicit request, an agent must choose among Greet, Elicit, Inform, Recommend, or Hold, where Hold means deliberately remaining silent. The difficulty is not only picking a helpful action but judging whether the current evidence is sufficient to act at all.
The paper calls its approach physically grounded because observations are converted into task-relevant retail state rather than into low-level physical dynamics. A customer comparing two products and inspecting labels can look identical to a customer browsing without a need, so the representation must preserve the cues that distinguish information seeking from independent exploration. The authors stress that a compressed intent label alone can discard exactly this evidence.
Splitting perception, prediction, and choice
The Proactive Intent World Model, or PIWM, decomposes the problem into three interfaces. See builds a perceptual basis from video: four evidence fields describe visible scene, engagement, gaze and attention, and body and hands, while stage, intent label, belief, desire, and intention express an inferred customer state. Foresee then predicts a one-step outcome for each candidate action, including next AIDA–BDI state, benefit, risk, and a relative reward on the interval from -1 to 1. Act compares these candidate–consequence pairs and selects an action.
The state design combines AIDA purchasing stages (Attention, Interest, Desire, Action) with BDI fields for belief, desire, and intention. The paper is explicit that the inferred fields are task interpretations, not observed facts; the evidence fields are retained so downstream modules can see what supported the interpretation. Hold is evaluated through the same Foresee interface as active interventions, so silence is a forecast alternative rather than an absence of choice.
The implementation uses a Qwen2.5-VL-7B-Instruct backbone with LoRA adaptation and trains the three tasks separately rather than as one monolithic video-to-action predictor. At inference, See constructs the state, Foresee scores every candidate from that same state, and Act compares the resulting pairs. Predictions are one step only and are not rolled forward into an imagined multi-step trajectory.
Structured inputs carry most of the gain
The central comparison uses the Full test set of 509 examples. Supplying predicted consequences to the decision step produced 63.65% action accuracy, while a direct-decision configuration that skips Foresee at inference reached 45.38%. Both configurations received the complete predicted state, so the difference isolates whether candidate consequences were compared.
The evidence ablation is sharper. Removing the four observable evidence fields and regenerating forecasts from the reduced state cut accuracy to 27.70%. Because this change alters both Foresee and Act inputs, the paper describes it as measuring their combined response to evidence removal rather than the effect on one module. The authors also report that performance is poor when the agent must extract information from raw video and decide directly, but improves substantially with structured inputs extracted and annotated from a professional retail perspective.
Accurate forecasts do not make good planners
Counterfactual prediction performs well in standalone evaluation, yet the paper reports that planning methods querying these forecasts at inference time degrade sharply. This is the study's most consequential negative result: a locally useful consequence prediction does not reliably improve action selection. The authors suggest the gap may reflect incomplete process understanding, uncertainty in fine-grained single-step outcomes, and insufficient joint modeling of scenes and temporal evolution.
A separate control isolates Hold. Withholding only the Hold forecast, while keeping the full state, the other four forecasts, the five-action candidate set, and the non-intervention instruction, reduced accuracy to 47.94%. Hold remains a legal choice in that condition, so the drop points to the value of forecasting silence itself rather than to removing the option. The paper also notes that Hold is the hardest action in structured-state evaluation, which it links to temporal awareness.
What the benchmark actually scores
GuidanceSalesBench supplies task-specific resources: 1,043 multimodal observation records for See, 2,001 state–action records for Foresee, 680 textual-state decision records for Act, and 3,054 examples in the Full collection, split into 3,054 training, 509 validation, and 509 test examples. The paper warns that these task-specific exports should not be read as counts of independent scenes or summed with Full.
Scoring differs by task. See requires both stage and intent to be correct over 209 examples, so a correct stage with a wrong intent earns zero. Foresee compares the predicted next AIDA stage with its reference over 309 state–action records; risk, benefit, reward, and generated BDI text are part of the consequence representation but are not additional terms in that accuracy. Act is scored over 245 textual reference-state examples, and Full over 509 test examples. A useful consequence need not change the AIDA label, so Foresee's next-stage score and Full decision accuracy measure different parts of the pipeline.
A separate Real set contains 418 sales-expert-annotated records from staged real-store scenes, excluded from training and reserved for evaluation. The consequence targets throughout are normative annotations rather than outcomes observed after real interventions, and the real-store material is staged rather than a covert deployment.
Where the evidence stops
The experiments cover single-decision prediction and selection with synthetic, normative supervision. Longer interactions and actual customer outcomes require other evaluation settings, and the paper states that its experiments do not establish customer benefit or deployment safety. Generalization across source scenes and real-store interactions is left to future work, as is tracing how state estimates and individual forecasts affect particular decisions.
The authors propose long-horizon interaction trajectories and temporal consequence supervision as the next step toward sustained reasoning and intervention timing. That proposal follows directly from the two weak points they document: forecasts that help locally but not in planning, and a Hold action that remains hard to get right.
The priority the paper establishes
For anyone building or evaluating a proactive retail agent, the paper's ordering of priorities is specific. Preserving explicit observable evidence matters more than compressing observations into an intent label: the ablation from full state to evidence-stripped state costs 35.95 percentage points of Full accuracy, far more than the 15.71-point gap between consequence-based and direct decision. Forecasting non-intervention belongs in the same comparison as active service, since losing only the Hold forecast costs 15.71 points.
The unresolved threshold is between prediction quality and decision quality. A model can score well on next-stage prediction and still fail when those forecasts are queried for planning, so standalone Foresee accuracy is not a sufficient proxy for whether an agent will act wisely. The paper's own framing supports treating intervention and non-intervention jointly and organizing observations under task knowledge, while reserving judgment on sustained assistance until multi-step trajectories and temporal supervision are evaluated.
Sources used in this answer
See Better, Foresee Better, Act Wiser: Physically Grounded Proactive Modeling and Decision Making
On a 509-example Full test set, consequence-based selection reached 63.65% action accuracy versus 45.38% for direct decision; removing four explicit evidence fields cut accuracy to 27.70%, and withholding only the Hold forecast cut it to 47.94%.