How we measured the error.

What we measured

We ran OnRubric, with the settings the site uses, on essays that people had already scored. For each set we wrote a rubric in our own words from that set’s scoring guide. For each essay we compared the middle of the range the model returned, before the site widens it, with the score the people gave. The average distance between the two, as a share of the rubric’s maximum, is the “average miss” printed on every page.

The essays

• 40 college essays from DREsS_New: argumentative essays by undergraduates writing in English as a foreign language. Each essay was scored by one instructor on content, organization and language, 15 points in total.
• 28 school essays from a research corpus of writing by English language learners in grades 8 to 12, scored by trained raters on six language traits (cohesion, syntax, vocabulary, phraseology, grammar and conventions).

Neither set was written for your course, and both come from students writing in a second language. No essay from either set appears on this site.

Results

The guess never reads the essay: for every criterion it predicts the average score of that set. On the school essays the guess came slightly closer than we did; on the college essays we were slightly closer. So the average miss on its own does not show that the estimate reads your paper well. On a 15-point rubric our miss is about 1.6 points.

Agreement with the people who scored

An estimate can keep its average miss small by giving every paper a similar score. Quadratic weighted kappa (QWK) measures essay-by-essay agreement, counts big disagreements more, and drops when every paper gets a similar estimate. A QWK of 0 means agreement at chance level, and 1 means full agreement.

On the 40 college essays our QWK was 0.36. A guess that gives every paper the same score has a QWK of 0. The DREsS authors report 0.30 for GPT-4o given the rubric alone, 0.43 with five scored examples, and 0.69 for the best model trained on their data (Table 3 of the paper, on their own test set). Our total falls between their two GPT-4o figures. By trait it is uneven: higher than their GPT-4o without examples on organization, lower on content and language. 40 essays leave a wide margin either way.

Where the estimate goes wrong

On the college essays it pulled scores toward the middle: essays that people scored low got estimates that were too high, and essays they scored high got estimates that were too low. On the school essays it was low across the board, at or below the people’s score on every essay, by 11.0 % of the maximum on average. If your draft is very strong, expect the estimate to understate it. Very weak drafts were rare in both sets, so the test says little about them.

Why every score is a range

When the model’s range for a criterion is narrower, the site widens it to about its midpoint plus or minus 15 % of that criterion’s maximum, rounded to half points and cut off at 0 and the maximum. We do not quote how often the real score fell inside the range: a guess with ranges as wide lands inside about as often, so that share describes the width, not the estimate.

What this test cannot tell you

• 68 essays is a small sample. A different set of essays could give different numbers.
• The 28 school essays are the part of a planned 40-essay sample that finished, so that set is not balanced by score.
• Each DREsS_New essay was scored by one instructor, so the data does not show how much two instructors would disagree about the same essay.
• Your instructor, your course and your rubric differ from the ones behind these scores.

What we measure next

We want to compare estimates with the grades students receive on their own rubrics. Sharing a grade will be optional (see Privacy). When there are enough of them, the result goes on this page.

Source

Yoo, H., Han, J., Ahn, S.-Y., & Oh, A. (2025). DREsS: Dataset for rubric-based essay scoring on EFL writing (arXiv:2402.16733v3; accepted to ACL 2025) [Preprint]. arXiv. https://arxiv.org/abs/2402.16733