570-student exam shows 'strict grader' prompts break open models; LoRA tuning restores parity

In an arXiv preprint (v1), the authors grade a practical Computer Vision exam for 570 dual-graded students under 171 configurations spanning closed and open-weights models, reporting that the best model reaches mean absolute error of 1.64 out of 35 points—below the 2.61 out of 35 that two human graders achieve against each other. The paper frames the real constraint as labor: one long-form exam in a large course costs hundreds of grader-hours, and qualified graders are scarce. The lure for institutions and vendors is obvious: swap scarcity and variability for automation that calibrates to humans. But the same study shows a short prompt that sounds like a supervisor’s instruction—“strict grader”—can push most open models out of a usable band or into refusal. This is a single-source, unaudited preprint on arXiv ; no one in the reported packet is on the record.

The best model beats human-human disagreement, but a two-sentence preamble collapses most open models

Against the 171 configuration sweep on the first exam, the headline number is tidy: a best-case mean absolute error (MAE) of 1.64/35, better than the 2.61/35 disagreement between paired human graders. Then comes the catch. A short “strict grader” preamble knocks 14 of 17 open-weights models out of the graded band (reported as MAE ≥ 8), with three stopping grading altogether. The paper attributes the break not to model size or the harsh tone, but to two credit-withholding sentences in the preamble; one sentence—“never give partial credit”—alone makes two of three probed models stop grading. In contrast, the closed flagships of three vendors shift calibration under the preamble but stay within the usable band. For buyers who assumed that “be stricter” is a harmless policy toggle, the mechanism matters: instruction wording is treated by many models as normative constraints on output rather than a reweighting of the same rubric. [S1]

The vulnerability replicates on a second exam, but the direction of error is exam-specific

On a second, independent Machine Learning exam from a different course with 1,038 dual-graded students, the study runs 162 further configurations. The “strict grader” preamble again knocks models around: it worsens ten models, pushing three into collapse and one into refusal. Yet it also improves seven models whose neutral prompts were over-marking. The authors’ point is not that a single persona is always harmful or helpful, but that sensitivity to instruction framing is real and that its direction is exam-specific. A procurement takeaway hides in that asymmetry: if your acceptance test is a single exam and a single house prompt, you can both overpay for a model that looks calibrated and then ship a policy tweak that collapses production performance. [S1]

For RFPs and SLAs, define the band, test refusals, and ban persona toggles as policy

Most evaluation pitches today sell “rubric-aligned” and “human-level” as static labels. The preprint demonstrates that both are conditional on the exact instructions packaged with the model, and that small changes to those instructions can change not only error magnitude but failure mode—into refusal. For buyers—universities, certification bodies, and the EdTech vendors supplying them—the operating change is contractual: define an acceptable error band relative to human-human disagreement on a held-out set, measure refusal rates explicitly, and treat persona or tone prompts as experimental features, not policy. An acceptance plan that only measures average error without a refusal ceiling will miss the “never give partial credit” failure the study surfaces. And for those opting for open weights, the test plan cannot rely on a single in-house prompt; it should include adversarial instruction variants to bound sensitivity before go-live. [S1]

Closed models hold up under harsh instructions, but LoRA fine-tuning gives open models a viable path

The closed flagships in the study stay inside the band under the strict preamble, albeit with calibration shifts. That tracks with vendor claims of instruction-following guardrails, but the preprint doesn’t name models or methods, so buyers won’t learn which specific combination of provider and prompt regime was resilient. More interesting for cost and lock-in is the remediation path the authors test: light LoRA fine-tuning. Training a single adapter on the two exams’ pooled ~3,900 graded examples brings five small open models to parity or better with a human grader in agreement with the grader pair, and sensitivity to the three harsh personas nearly vanishes (≤ 0.32 MAE). For procurement, that implies a forked strategy. If you standardize on closed models, write the persona-robustness and refusal ceilings into the SLA and require providers to disclose calibration drifts when policy prompts change. If you prefer open models, reserve budget and data-governance capacity for collecting and labeling a modest corpus and for maintaining fine-tuned adapters as the rubric or intake format evolves. The paper’s open release of an anonymised dataset, ablation grid and pipelines reduces the barrier, but it doesn’t remove your obligation to maintain your own calibration data under your data policies. [S1]

This is not just an education story; it’s a warning for any automated evaluator sold on prompt policy

The use case here is exam grading, but the mechanism—persona- or instruction-induced calibration shift and refusal—is the same one being pitched into code reviews, policy compliance checks, customer-service QA and internal knowledge tests. In those domains, buyers are often told they can flip a strictness switch in a system prompt to align with a brand voice or regulatory posture. The preprint shows you can also flip your system into non-grading or over-penalizing behavior if the underlying model treats that switch as a hard constraint. If you are embedding an automated evaluator into a workflow with appeals (student regrades, compliance exceptions, QA escalations), your contract should make explicit how refusal cases are routed and resolved, and how often they are expected to occur under the accepted instruction set. [S1]

The skeptic’s read: two CS exams are not the world, and arXiv is not peer review

Skeptics will point out that both exams are in computer science and may not generalize to humanities grading or to free-form creative work, and that the preprint is not peer reviewed. The authors themselves note that prompt sensitivity’s direction is exam-specific: seven models actually improved under the strict preamble when the neutral prompt over-marked. A cautious procurement response is to treat the paper as a test plan, not a universal claim: adversarially test your chosen model on your rubric, your format, and at least one unrelated exam or QA set before production. The release of the anonymised dataset, full ablation grid, and pipelines is useful here because it allows independent replication and lets vendors run the same protocol on their stacks; it does not, by itself, guarantee your domain will behave the same way. [S1]

What changes now for sellers and buyers of AI grading and evaluation tools

For sellers, a “human-level” headline number is no longer enough. Expect RFPs to request: MAE against dual-human graders on held-out sets; refusal rates under neutral and harsh instruction variants; and a demonstration of stability when the instruction preamble explicitly withholds partial credit. Vendors leaning on prompt engineering as their primary calibration tool will be pushed toward either fine-tuning (as in the LoRA result) or toward more conservative closed models with instruction-safety features—and will need to price the associated evaluation operations. For buyers, the economic calculation shifts from “grader-hours saved” to “evaluation-ops time invested.” You may replace hundreds of manual hours with a smaller, ongoing commitment to dataset curation, calibration checks and prompt-change reviews. Budget that in, and set decision rights: procurement and legal should sign off on any policy prompt changes, because the study shows they can materially affect system behavior. [S1]

Looking ahead six months, watch for vendors to publish persona-robustness evaluations and to add refusal ceilings to their SLAs, for universities to require a held-out exam evaluation in procurement, and for at least a handful of open-weight providers to ship LoRA adapters or recipes tuned to common grading rubrics. If those signals do not materialize, it will suggest sellers are still treating instruction fragility as a support issue rather than a core product property—and buyers should respond by writing their own tests into contracts. [S1]