Mentioned content is not the same as active intent
The paper's central distinction is between what was said and what remains in effect. It calls the requirements that still govern the answer the active intent. A dialogue can contain a proposal that was floated and then rejected, or a requirement that was replaced, without those statements changing the task. The model's job is to answer against the active intent, not against everything that appears in the context.
The authors name the failure mode mentioned-as-in-effect confusion: conversational content is treated as an active requirement even after it has been rejected or superseded. This is a claim about the pattern behind the errors, and the paper supports it by inspecting the new errors that appear under rejection and revision relative to the uninterrupted dialogue, where those errors often contain rejected or superseded content.
Four matched conditions isolate the effect of a decision
Intent-Eval builds four multi-turn conditions from the same source task. Original discloses the task across turns with no inserted event. Neutral inserts two clarification questions about the selected requirement without proposing a new value. Retained and Revised share the identical proposal, then diverge: Retained rejects it, Revised accepts it. Only Revised changes the reference answer.
Because Retained and Revised share the same proposal text and differ only in the user's decision, any accuracy gap between them cannot be explained by the proposal's wording. Original, Neutral, and Retained are all scored against the original reference answer, while Revised is scored against the revised one. The benchmark also includes single-turn controls that state the whole task in one message, which separate multi-turn effects from the difficulty of the task itself.
Three separate sources of accuracy degradation
The paper reports three patterns across eight models. First, dispersed disclosure: splitting a task across turns lowers mean accuracy by 36.30 percentage points relative to the single-turn control, even in Original, where nothing is interrupted. Second, conversational overhead: adding neutral clarification that does not change the task costs a further 4.43 points. Third, decision-induced confusion: proposing a change degrades performance whether the user rejects or accepts it.
The third pattern is the one the title question targets. In multi-turn dialogue, Retained averages 8.06 points below Original and Revised averages 5.55 points below Original. The authors also note that decision-induced confusion appears in single-turn controls, meaning a proposal and decision can hurt performance even without multi-turn interaction. These are mean drops across the evaluated models and domains, not a claim about any single task.
Degradation deepens as the interaction continues
The benchmark extends dialogues in two ways. Accumulation repeats clarification or rejection blocks, so the requirement under discussion stays fixed while more rejected proposals pile up. Persistence places clarification blocks after a decision has already been made, so the active task is unchanged at every depth.
Accuracy tends to fall as users ask for clarification over more turns or propose and reject more changes. The paper's phrasing is that degradation can deepen or persist as interaction continues. This matters for reading the headline numbers: the 8.06 and 5.55 point gaps are averages over the base conditions, and the extended dialogues show the problem is not self-correcting as the conversation grows longer.
What Intent-Eval actually measures
Intent-Eval contains 414 source tasks drawn from four domains: Actions for tool calling from BFCL (104 sources), Code for program synthesis from HumanEval and LiveCodeBench (100 sources), Database for text-to-SQL from Spider (107 sources), and Math from GSM8K (103 sources). Each source yields four multi-turn conditions and four single-turn controls, giving 3,312 evaluation instances per model. Multi-turn dialogues average 7.14 turns, ranging from 3 to 14.
Each source task has exactly one requirement revised, with reference answers for the original and revised states. Revisions are constrained to be unambiguous, atomic, and verifiable: the change must specify exactly which requirement is replaced, affect only that requirement, and produce a reference answer or output contract a task-specific verifier can check. Only the final response is scored. The benchmark therefore measures whether a model's final answer matches the active task state, not whether it acknowledges the decision in its wording.
Intent-OPSD gives the teacher the task that remains in effect
Intent-OPSD is a decision-conditioned on-policy self-distillation framework. Teacher and Student start from the same model. The Teacher is frozen and receives the complete single-turn task matching the user's decision, while the Student receives the full multi-turn dialogue and samples its own final answer on-policy. At each final-answer position, both policies score the same Student-generated prefix, and the loss is the Jensen-Shannon divergence between their next-token distributions.
The four condition losses (Original, Neutral, Retained, Revised) are averaged equally, and only the Student is updated. Intermediate replies stay in the dialogue history but receive no loss. The key design choice is matching the Teacher's single-turn prompt to the decision: the original single-turn task shares active requirements with Original, Neutral, and Retained, while the revised single-turn task matches Revised. At inference the Student responds directly from the dialogue, without the Teacher or the complete-task prompt.
Where the gains are largest and where they are not
Intent-OPSD improves mean accuracy over the corresponding base models by 10.81 percentage points across four models and four domains. The paper reports the largest gain in Actions, 21.46 points over Base, where selecting the correct call arguments can fix the output. Math and Database are described as similarly likely to benefit from tracking which numerical premises and query constraints remain active. Code shows smaller gains, which the authors attribute to its demands on program structure and logic: even a model that tracks the active requirement correctly can still fail on implementation.
The paper is explicit that the method helps when intent tracking is the bottleneck, not when the model cannot solve the task at all. It also reports that mean Retained accuracy under Intent-OPSD is similar to matched supervised fine-tuning, so handling rejected changes remains the harder half of the problem. That asymmetry is the paper's own stated limitation and the clearest signal about where further method work is needed.
What the benchmark does not yet cover
Intent-Eval focuses on decisions about one selected requirement per source task. Extending coverage to multiple interacting requirements and more varied user behavior is left as future work, so the measured gaps describe single-requirement revisions rather than the tangled requirement changes of real conversations.
The training setup also relies on complete active-task specifications supplied by the benchmark construction. Those specifications support the Teacher during training, while the trained Student uses only the dialogue at inference. Applying the approach to natural conversations would require reconstructing active requirements from user messages and decisions, with consistency checks and human review for ambiguous cases. The reported gains therefore apply to settings where the active task state is known during training, not to arbitrary open-ended dialogue.
Sources used in this answer
You Changed Your Mind, The Model Didn't: Demystifying Intent in Multi-Turn Dialogue
Across 414 tasks and eight models, merely mentioning a rejected change degrades task execution: Retained averages 8.06 percentage points below the uninterrupted Original dialogue and Revised 5.55 points below, consistent with mentioned-as-in-effect confusion. Intent-OPSD, a decision-conditioned on-policy self-distillation method, improves mean accuracy by 10.81 points over base models.