Practice scores and knowledge tests are not measuring the same skill
Practice questions usually reward getting to an answer, and AI can help with exactly that step. Medical students who used AI-generated practice questions saw perceived utility and performance benefits in that practice setting [3]. A personalized AI learning platform in Chinese higher education also raised post-assessment scores across five subjects, with gains ranging from about 13% to 23% depending on the subject [5]. Those results show AI can improve scores on the tasks it is used for.
A knowledge test asks whether the student can produce the understanding without the tool in the room. None of the studies here directly compares the same students on AI-assisted practice versus an unaided knowledge test, so the practice-versus-knowledge gap is not measured in these papers. What the evidence does support is narrower: AI-assisted practice can improve practice performance, and that improvement should not be read as proof of durable knowledge.
Why the help can stop at the practice screen
One reason is that AI tools often act as answer providers rather than as tutors that force the student to reconstruct the reasoning. The medical-student study examined AI-generated practice questions as a learning adjunct and reported perceived utility and performance outcomes, but it did not test whether students could solve similar problems without the tool [3]. If the tool supplies the next step, the practice score can rise while the underlying skill stays where it was.
A second reason is that the measured outcome depends on what the tool was designed to improve. AI-based flashcards were studied for their effect on EFL reading comprehension, a specific outcome, not for general knowledge transfer [2]. A personalized platform improved post-assessment performance in five subjects [5], but those post-assessments were tied to the same platform-supported instruction. Improvement on the taught task is not the same as improvement on an independent knowledge test.
Code comprehension is especially hard to fake with a hint
Code comprehension asks a reader to trace what a program will do, not just to produce a working line. An AI assistant can generate code or suggest a fix, but the student still has to follow the logic to answer a comprehension question. None of the studies here measured code comprehension directly, so this part of the question is not settled by the available evidence. The closest relevant result is that AI integration improved computer science post-assessment scores by 18.4% in one experiment [5], which shows AI can affect a technical subject's measured performance without telling us whether students could read unfamiliar code on their own.
What would make the practice gains carry over
The teacher-focused studies point to a condition rather than a technique. In a survey of 384 English teachers, perceived ethical commitment and competence in AI-related pedagogy strongly predicted teachers' intention to use AI ethically, and that intention was the strongest predictor of student learning outcomes, ahead of reported classroom practices [4]. That is a correlation, not proof that intention causes learning, but it suggests that how the tool is framed and used matters as much as whether it is used.
The French-teacher survey in Lagos State adds a practical constraint: adoption of pedagogically rich AI tools was low, with only 6.7% of teachers using AI chatbots often or very often, while 68.3% often used a basic translation tool, and limited computer and internet access was rated the most significant barrier [1]. A tool that is used mainly as a translator is unlikely to build the kind of understanding a knowledge test or code-comprehension task requires. The gap between practice and knowledge is therefore partly a gap between using AI to get answers and using AI to practice retrieval and reasoning without it.
Sources used in this answer
FRENCH TEACHERS' USE OF ARTIFICIAL INTELLIGENCE AS A TOOL FOR ENHANCING THE SECONDARY SCHOOL FRENCH STUDENTS' PERFORMANCE IN LAGOS STATE
In a survey of 60 French teachers in Lagos State, only 6.7% used AI chatbots often or very often, while 68.3% often used Google Translate, and limited computer and internet access was the most significant adoption barrier.
The Effects of AI-Based Flashcards on EFL Students' Reading Comprehension: A Pre-Experimental Study in Indonesian Higher Education
A pre-experimental study found that AI-based flashcards were associated with higher reading comprehension scores among EFL students in Indonesian higher education.
Medical students' perceptions of AI-generated practice questions as learning tools
Preclinical medical students reported perceived utility and performance benefits from AI-generated practice questions, but the study did not test unaided knowledge or code comprehension.
Modeling School Teachers’ Ethical Use of AI Tools in English Instruction and Its Impact on Student Learning: A PLS-SEM Approach
Among 384 English teachers, perceived ethical commitment and AI-related pedagogical competence predicted intention to use AI ethically, and that intention was the strongest predictor of student learning outcomes, ahead of reported classroom practices.
Integration of modern technologies in higher education on the example of artificial intelligence use
An experiment with 356 Chinese higher-education students found that a personalized AI learning platform raised post-assessment scores across five subjects, including an 18.4% gain in computer science.