Does transparency improve students' perceptions of AI autograders?

Transparency lifted perceived accuracy, not exam trust

The study's clearest result is narrow. Students who received extra information about the autograder rated its grading accuracy at a mean of 0.90 on a scale running from -2 (disagree) to 2 (agree), compared with 0.59 in the control group. A Mann-Whitney U test put that difference at p = 0.027.

That gain did not carry over to every attitude. Willingness to have the autograder grade exams showed no meaningful difference (treatment mean 0.78, control 0.84, p = 0.80), and neither did liking the assignment's autograders (0.73 versus 0.77, p = 0.77) or overall satisfaction (0.89 versus 0.78, p = 0.61). Perceived helpfulness for improving plain-English code explanations moved in the expected direction but only marginally (0.85 versus 0.66, p = 0.083).

A one-hour activity with a new grader

Participants came from an introductory computer science course for non-majors at a large public university in the United States, recruited in the second-to-last week of the semester for a small amount of extra credit. They were randomly assigned to conditions: 81 in the treatment group and 74 in the control.

The activity centered on Explain in Plain English (EiPE) questions, which ask students to describe in ordinary language what a piece of code does. These were graded by an autograder built on OpenAI's GPT-4 and modeled on a few-shot LLM-based autograder. This mattered for interpretation: students already knew a different, bigram-based grader in the course that returned only right-or-wrong answers, whereas the study's grader gave written feedback on how to improve.

The treatment group read a transparency statement and answered a comprehension quiz about it. The statement told students the graders were developed from prior student data, validated by experienced course staff and researchers, calibrated on realistic responses, and comparable in accuracy to human graders. It also explained that grading follows the overall meaning of a response rather than specific keywords, and advised students to reword answers marked incorrect. Each learning-activity question added the number of real student submissions used for testing and the accuracy achieved, which varied by exercise. The control group was told only that an AI autograder was used.

The accuracy figures came from prior validation

The accuracy numbers shown to the treatment group were not invented for the survey. Before the study, the autograder was prompted to act as a computer science teacher, given a specific EiPE question with correct and incorrect example answers, and then validated on real student responses. The authors aimed to keep accuracy above an 80% threshold and to validate on at least 20 student responses.

This design means the transparency intervention conveyed genuine measured performance of the grader on prior data, not a claim about how it would score each student's current answer. The paper does not report a single overall accuracy figure for the study's own grading; the numbers students saw varied by exercise.

High baseline trust may have compressed the effect

The authors offer an explanation for why transparency failed to shift willingness to be graded by the autograder. They suggest the null result may stem from higher measured student trust toward autograders in this study than in prior work in the field. If trust was already relatively high, there was less room for a transparency statement to raise it.

That is the authors' interpretation, not a tested mechanism in the trial. The study did not manipulate baseline trust or compare trust levels directly against earlier cohorts. What the data do show is a split between believing the grader is accurate and accepting it for high-stakes grading.

Students separated feedback from final grading

Across both conditions, students preferred immediate autograder feedback to waiting a week for human grading: the relevant item averaged -0.02 in the treatment group and -0.05 in the control, with p = 0.98, indicating no difference between groups rather than a strong preference in either direction. Students also expressed willingness to use the autograder for exam preparation (1.07 treatment, 0.95 control, p = 0.37), but were less enthusiastic about using it to grade exams.

The open-ended responses sharpen this distinction. A theme the authors highlight is conditional trust: students saw the autograder as fine for practice but wanted human verification when grades were at stake. One quoted treatment student put it as useful for feedback but not something to rely on. The authors note that in this course, answers to similar questions using a different natural-language autograder were validated by course staff during exams, and that familiarity may have shaped these comments.

Transparency increased willingness to comment

An unexpected finding came from the open-ended survey questions. For three of the four questions, treatment students wrote substantially more characters than control students. On the final comments question, the difference in character counts was statistically significant (p = 0.043). The exception was the question asking about negative experiences, where the two groups wrote almost the same amount.

The authors describe this as preliminary but suggestive: transparency may have made students more willing to engage with and comment on the autograder. The analysis reports theme counts and percentages by group, but the paper does not establish why transparency would increase comment length.

A short activity cannot settle semester-long trust

Several boundaries limit how far these results travel. The intervention was a single short learning activity in one introductory non-major course, and the autograder differed from the one students normally used, so students were responding partly to a novel tool. The post-test performance comparison found no significant difference between groups, and the authors state they were not concerned with reconciling pre- and post-test grades across multiple graders because their interest was perceptions, not performance.

The authors themselves call for longitudinal work to see whether this kind of intervention has a larger effect over a semester rather than in a one-hour activity, and for research on how autograder interactions affect learning and retention rather than only perceptions.

Transparency buys credibility, not consent for high-stakes grading

The practical priority this study supports is specific: transparency about an autograder's validation data and accuracy is a plausible way to improve students' belief that its grading is accurate, and possibly their sense that its feedback helps. It is not, on this evidence, a way to win acceptance for using the autograder as the final grader on exams.

The gap between those two outcomes is the paper's most useful signal. Students in this trial were already relatively trusting, preferred fast feedback, and still drew a line at high-stakes grading without human validation. Anyone deploying an AI autograder should treat perceived accuracy and consent for graded use as separate targets, and should expect that explaining the system may move the first without moving the second.

Sources used in this answer