Why a smooth, confident answer feels trustworthy
When an AI response reads fluently and sounds sure of itself, students tend to treat that confidence as a signal of accuracy. In one set of findings, 61% of students found AI answers convincing as such, and a confident tone was cited by 38% as a reason for that conviction [1]. The fluency does real work here: it removes the small hesitations and hedges that might otherwise prompt a reader to pause and ask, "is this actually right?"
This is the core of what researchers call automation bias—the tendency to defer to a machine's output as if it were an objective authority. One study framed the effect around students treating AI as an "objective" machine, and found that this deference was not simply a matter of being uninformed [3]. The confidence of the presentation and the perceived objectivity of the tool combine to make checking feel unnecessary.
Knowing about the problem doesn't reliably fix it
A natural assumption is that students who understand AI well will be the ones who check sources. The evidence complicates that. One study reported that AI literacy did not protect against automation bias [1], meaning that even students with a good grasp of how these tools work can still be swayed by a confident answer.
What seems to help is not general knowledge but a deliberate habit of pausing. Work on critical agency describes the risk that heavy reliance on AI tools encourages passive learning, and points toward self-awareness of automation bias and "conscious checks" as the counterweight [2]. In other words, the useful skill is catching yourself in the moment of acceptance and choosing to verify, rather than assuming that understanding the technology will make you skeptical automatically.
Can the interface itself prompt a check?
If fluency is part of the problem, one response is to build in a deliberate interruption. A design concept called Epistemic Friction aims to do exactly that—introducing small points of resistance meant to encourage students to verify AI-generated information rather than absorb it [1]. The idea is to disrupt the smooth glide from reading to believing.
The evidence here is at the level of a proposed design concept rather than a demonstrated fix, so it is best read as a promising direction rather than a proven solution [1]. What the studies do support is the underlying diagnosis: fluent, confident presentation shapes students' credibility judgments, and that is the lever any intervention would need to work on [1][3].
Where students land when a source disagrees
Checking sources matters most when the AI and another source conflict. The available findings suggest students are not uniformly loyal to the machine: 86% said they would side with a non-AI source in a disagreement [1]. That is a meaningful counterweight to the picture of blind trust, and it suggests the pull of a confident AI answer is not absolute.
Still, stated willingness to side with a human or non-AI source is not the same as actually going to look. The gap between saying you would check and actually checking is where automation bias does its quiet work, and the research so far describes the tendency and proposes ways to counter it more than it measures how often students verify in practice [1][2][3].
Sources used in this answer
Designing Epistemic Friction to Foster Verification in Students' Use of Generative AI
Introduces a design concept called Epistemic Friction to encourage students to verify AI-generated information, and reports that 61% of students found AI answers convincing, 38% cited a confident tone, 86% would side with a non-AI source in a disagreement, and AI literacy did not protect against automation bias.
From Algorithmic Output to Human Understanding: Building Critical Agency and Trust in AI-Mediated Research and Education
Argues that reliance on AI tools can encourage passive learning and automation bias, and points to self-awareness of automation bias and conscious checks as ways to build critical agency and trust.
Evaluating the Validity of Arguments Presented in Human vs. AI Interfaces
Examines whether the authoritative tone of AI-generated text and treating AI as an objective machine (automation bias) influence undergraduate students' judgments, though it does not clearly establish whether students verify sources.