Why a gain today is not the same as learning that lasts
A tutoring session can lift performance in the moment for reasons that have little to do with durable learning: the AI is right there, prompting the student, breaking the problem into steps, and keeping attention on the task. That is genuinely useful, but it is a different thing from a student being able to solve the same kind of problem alone a month later. Testing retention means removing the support and waiting, then measuring again.
This distinction matters because the studies that report the clearest wins tend to measure outcomes close to the tutoring itself. In an exploratory randomized trial across five UK secondary schools, students working with a pedagogically tuned AI model performed at least as well as students chatting with human tutors on every learning outcome measured, and were 5.5 percentage points more likely to solve novel problems on later topics [5]. That is a meaningful result about transfer to new problems, but the measurement still sits inside the study period rather than after a long delay.
The one experiment that actually tested retention
The two-year Khanmigo school experiment is the study that speaks most directly to the question, because it included measures of retention rather than only immediate performance [2]. It compared AI tutoring against existing daily remedial mathematics sessions and against ordinary practice and math software without AI tutoring, which gives a genuine comparison group rather than just a before-and-after snapshot [2].
What the available summary does not give is the size or direction of the retention result. The abstract notes that retention carries a transparent cost, which hints that sustaining gains over time involves a real trade-off rather than a free improvement [2]. Until the full results are read, the honest answer is that delayed outcomes were measured in this experiment, but the evidence here does not establish that the engagement gains held up.
What the other studies can and cannot tell us
A middle-school experiment built around mastery-based practice tested an AI tutor embedded directly beside each math exercise, with guardrails on what the tutor could say [3]. This design targets exactly the kind of supported practice that might build lasting skill, but the study does not report delayed retention outcomes, so it cannot answer whether the gains stuck [3].
Work comparing two ChatGPT models on trigonometry tutoring focused on instructional quality and adaptivity, judging how well the models taught rather than how much students retained afterward [4]. Similarly, research using learning analytics to map the division of labor between human and AI in generative AI math tutoring examines how those roles play out during tutoring, not whether learning persists once the tutoring ends [1]. These are useful for understanding what good AI tutoring looks like in the moment, but they leave the retention question open.
How to judge the next claim you see about AI tutoring
When you encounter a headline that AI tutoring improves math learning, the useful question is what was measured and when. A study reporting that students solved more problems correctly during or immediately after sessions is telling you about supported performance; a study that retests students after a gap without the tool is telling you about retention. The UK trial's finding that students did better on novel problems in subsequent topics is closer to the second kind of evidence than a same-session score, because it involves applying learning to something new [5].
For now, the practical reading is that AI math tutoring has credible evidence of immediate and near-term benefit, including against human tutors in one trial [5], and that at least one long school experiment measured retention but does not, in the evidence available here, confirm that the gains persisted [2]. The question is a good one and remains genuinely open rather than settled in either direction.
Sources used in this answer
Using Learning Analytics to Unpack Human–AI Roles in Generative AI Math Tutoring
The paper examines human and AI roles in generative AI math tutoring through learning analytics but does not address whether engagement gains persist when retention is tested after a delay.
One click away: AI tutoring with Khanmigo in a two-year school experiment
A two-year school experiment with Khanmigo in mathematics compared AI tutoring against remedial sessions and ordinary math software, and included measures of retention, though the available summary does not report the retention result.
Making AI tutoring productive: Evidence from a mastery-based math practice experiment
A middle-school mastery-based math practice experiment evaluated an AI tutor embedded beside exercises with guardrails, but did not present evidence on delayed retention outcomes.
A comparative analysis of ChatGPT's trigonometry tutoring capabilities: Progressing from model 3.5 to reasoning o1
A comparison of two ChatGPT models on trigonometry tutoring assessed instructional quality and adaptivity, not retention or long-term learning outcomes.
AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms
An exploratory RCT in five UK secondary schools found AI-supported tutoring produced learning outcomes at least as good as human tutoring, with students 5.5 percentage points more likely to solve novel problems on subsequent topics, but it did not assess retention after a delay.