When the Evidence Was the Exam: A Statistics Course Put Its Own Grades on Trial

The appeal arrived in October, in the format the university requires: a signed form, a contested rubric score, a request for review. In most courses, it would have been resolved quietly — a meeting, an explanation, perhaps a point adjustment. In Dr. Mariko Hase's second-semester statistics course, it became the syllabus.

Rather than adjudicating the appeal herself, Hase distributed it to the class. Thirty-one students were asked to treat the complaint the way they would treat any dataset: clean it, code it, test it, and report what they found. The exercise, now in its second iteration, has become one of the most closely watched pedagogical experiments on campus — and a case study in whether students can meaningfully audit the institutions that grade them.

An Appeal Treated as a Data Point

The original grievance was narrow. A student argued that two graders had scored identical short-answer responses differently, costing a partial credit on an exam question. Hase's instruction to the class was simple: verify the claim, and in doing so, examine every assumption the course makes about how its grades are produced.

Students formed three working groups. One reconstructed the original grading, scoring a sample of forty anonymized exam responses independently and comparing results. A second group coded the rubric itself, cataloging every subjective term — "adequate justification," "clear reasoning" — that lacked operational definitions. The third group interviewed, with consent and anonymity protocols, graders from previous semesters about how they had interpreted the standards in practice.

"The students were, in effect, conducting an inter-rater reliability study on their own course," Hase says. "That is exactly the kind of analysis they would perform on any dataset in a research methods sequence. The only difference was that the subject was themselves."

What the Audit Found

The findings, presented in a public session at the end of the term, were more uncomfortable than anyone anticipated. Independent re-grading of the forty-response sample produced a Cohen's kappa of 0.62 — a level of agreement generally described as substantial but far short of the consistency the department had assumed. Nine of the forty responses would have received different scores depending on the grader.

The rubric audit identified eleven subjective phrases with no defined criteria. The most consequential finding concerned partial credit: graders consistently awarded more points to answers that matched the structure of the instructor's model solution, even when the underlying reasoning was equivalent. Students called this "template bias," and documented it across three prior semesters of archived responses.

"We proved the appeal was correct," one student wrote in the final report, "but the appeal was also too small. The problem was not one score. It was the instrument."

Rewriting the Policy

The audit's final deliverable was not a grade change — the original appeal was resolved early, in the student's favor, under standard procedure. It was a proposed revision to the course's assessment policy, drafted by students and adopted by the department after a semester of review.

The new policy requires double-blind grading of all exam questions worth more than ten percent of the final grade, operational definitions for every rubric term before each exam is scored, and a published reliability check each semester. Perhaps most notably, it establishes a standing audit committee of enrolled students who review a random sample of graded work each term.

"The students did not just find error," says Dr. Alan Reyes, chair of the mathematics department, which approved the policy. "They designed a system to catch error continuously. That is a harder thing to do, and it is why the department adopted it rather than simply thanking them."

A Model With Limits

The experiment has drawn attention beyond Atsugi, including inquiries from assessment researchers at two US universities studying student partnership models. But Hase is cautious about generalizing. The audit worked, she argues, because the discipline itself supplied the methodology — students already knew how to measure reliability and detect bias. A literature course could not replicate the exercise as easily.

There are limits of power, too. The audit committee can flag inconsistencies but cannot change grades directly; final authority remains with the instructor and the department. Critics on campus have asked whether students, whose own scores are affected, can audit a system they have a stake in. Hase's answer is that every auditor has a stake — the point is disclosure, not detachment.

"We ask students to trust a grading process they never see," she says. "This course asks the opposite question: what if the process had to survive the people it measures?"

Whether student-led course audits spread will depend on whether other departments are willing to submit their own instruments to the same scrutiny. For now, the statistics course has made one thing clear: the most rigorous examination some students will take may be the one they administer themselves.