SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework

# Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework
Source: [https://arxiv.org/html/2607.00274](https://arxiv.org/html/2607.00274)
Shayan Peyghambari Oskoui, Norah Almousa, Zhaoyi Joey Hou, Carolina Gustafson, Gayle Rogers, Raquel Coelho, Diane Litman, Xiang Lorraine Li University of Pittsburgh \{shayan\.p, xianglli\}@pitt\.edu

###### Abstract

Effective writing feedback is among the strongest drivers of student learning, yet producing it at scale is labor\-intensive\. LLMs offer a natural path to scaling writing support, but two gaps stand in the way: few public corpora capture how instructors actually deliver feedback in real classrooms, and no reliable method measures whether generated feedback aligns with what an instructor would write\. We address both\.Seforais a public corpus111[https://github\.com/ShayanPey/SEFORA](https://github.com/ShayanPey/SEFORA)pairing instructor inline feedback with assignment prompts, rubrics, scores, and multi\-draft revisions across various college writing genres, comprising 564 drafts and 8,240 instructor annotations\.UniMatchis a reference\-based evaluation framework for open\-ended generation: it segments feedback into feedback units, scores their semantic correspondence under instructor\-derived criteria, and aligns them via optimal matching to yield interpretable precision, recall, and F1\. Across 74 experimental configurations spanning multiple LLMs, no setting exceeds0\.40\.4F1\.UniMatchreveals that models struggle to identify the feedback instructors would prioritize, and performance degrades as models generate more\.

SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework

Shayan Peyghambari Oskoui, Norah Almousa, Zhaoyi Joey Hou,Carolina Gustafson, Gayle Rogers, Raquel Coelho,Diane Litman, Xiang Lorraine LiUniversity of Pittsburgh\{shayan\.p, xianglli\}@pitt\.edu

## 1Introduction

Feedback plays a vital role in student learning in writing\. It helps students correct misunderstandings and refine how they apply knowledgeAheaet al\.\([2016](https://arxiv.org/html/2607.00274#bib.bib86)\); Banihashemet al\.\([2024](https://arxiv.org/html/2607.00274#bib.bib87)\), and it is consistently identified as one of the strongest influences on learning and achievementHattie and Timperley \([2007](https://arxiv.org/html/2607.00274#bib.bib88)\)\. But effective feedback is not one\-size\-fits\-all\. Students report that the most useful feedback is specific to their own writingLipnevich and Smith \([2009](https://arxiv.org/html/2607.00274#bib.bib55)\), and effective feedback accounts for the draft’s position within the revision processCarless and Boud \([2018](https://arxiv.org/html/2607.00274#bib.bib84)\); poorly targeted feedback can even be detrimentalKluger and DeNisi \([1996](https://arxiv.org/html/2607.00274#bib.bib83)\)\. In writing instruction, producing such feedback is labor\-intensive, and its cost at scale discourages the sustained practice effective instruction requiresApplebee and Langer \([2011](https://arxiv.org/html/2607.00274#bib.bib82)\); Graham \([2019](https://arxiv.org/html/2607.00274#bib.bib81)\)\. This opens a natural opportunity for NLP: systems that generate useful feedback on drafts could help scale writing support\.

![Refer to caption](https://arxiv.org/html/2607.00274v1/x1.png)Figure 1:Overview of theUniMatchevaluation pipeline\. For each paragraph, LLM\-generated feedback is segmented into units and compared with instructor feedback units\. The resulting semantic similarity scores are used to compute an optimal matching between instructor and model feedback units, producing the final evaluation metrics\.Progress on this problem is constrained by two bottlenecks\. First, few public datasets preserve how instructor feedback is actually delivered in coursework \(Table[1](https://arxiv.org/html/2607.00274#S2.T1)\): multifaceted comments \(often addressing several points at once\) anchored to specific spans \(a paragraph, sentence, or word\) and interpretable alongside the assignment prompt, rubric, and revision history\. Some resources substitute structured labels \(error tags or analytic scores\) for formative commentaryCrossleyet al\.\([2024](https://arxiv.org/html/2607.00274#bib.bib71)\); Mathias and Bhattacharyya \([2018](https://arxiv.org/html/2607.00274#bib.bib67)\); Dahlmeieret al\.\([2013](https://arxiv.org/html/2607.00274#bib.bib54)\); Leeet al\.\([2015](https://arxiv.org/html/2607.00274#bib.bib56)\), restrict coverage to a single prompt or narrow genreKashefiet al\.\([2022](https://arxiv.org/html/2607.00274#bib.bib72)\); Zyskaet al\.\([2026](https://arxiv.org/html/2607.00274#bib.bib23)\), or forgo expert annotation for crowd\- or model\-generated feedbackBehzadet al\.\([2024](https://arxiv.org/html/2607.00274#bib.bib22)\)\. Without datasets that capture feedback as instructors deliver it, evaluating whether LLMs produce what an instructor would write remains out of reach\.

Second, feedback evaluation is difficult for three reasons\. \(i\) Reference\-free evaluation is conceptually appealing, but ‘good’ writing feedback has no single operational definition, as its quality depends on many interacting dimensions weighted differently across raters and settingsPearson \([2022](https://arxiv.org/html/2607.00274#bib.bib8)\); Stahlet al\.\([2024](https://arxiv.org/html/2607.00274#bib.bib9)\); LLM\-as\-judge proxies inherit this ambiguity while adding biases of their ownZhenget al\.\([2023](https://arxiv.org/html/2607.00274#bib.bib10)\); Deutschet al\.\([2022](https://arxiv.org/html/2607.00274#bib.bib11)\)\. \(ii\) Reference\-based evaluation mitigates this by comparing generated feedback against instructor feedback, but existing metrics fall short: n\-gram overlap is unreliable for open\-ended generationNovikovaet al\.\([2017](https://arxiv.org/html/2607.00274#bib.bib13)\); Reiter \([2018](https://arxiv.org/html/2607.00274#bib.bib12)\), and embedding\-based metrics, while less surface\-bound, correlate only weakly with human judgmentsLiuet al\.\([2016](https://arxiv.org/html/2607.00274#bib.bib14)\), as we also find in our setting \(§[4\.2](https://arxiv.org/html/2607.00274#S4.SS2)\)\. \(iii\) A deeper, often overlooked issue is granularity: a feedback message typically bundles several distinct*feedback units*Yenet al\.\([2020](https://arxiv.org/html/2607.00274#bib.bib48)\); Zouet al\.\([2024](https://arxiv.org/html/2607.00274#bib.bib47)\)– self\-contained statements about specific aspects of the writing – where even a single sentence may contain several \(e\.g\., ‘This paragraph is well\-written, but try to make it more concise’\)\. Holistic comparison cannot tell which observations are recovered or missed, so reliable evaluation must compare feedback at the unit level and by its underlying meaning\.

We address both\.Sefora\(StudentEssays withFeedback Corpus\) is a public corpus pairing authentic instructor\-authored span\-anchored feedback with assignment prompts, rubrics, analytic scores, and multi\-draft revisions across diverse college writing genres\.UniMatchis a reference\-based framework that evaluates LLM feedback at the unit level rather than holistically, segmenting feedback into feedback units and aligning them via optimal matching under instructor\-derived similarity criteria; the resulting similarity\-weighted F1captures both what the model omits and what it over\-generates \(Figure[1](https://arxiv.org/html/2607.00274#S1.F1)\)\. Together they enable systematic study of LLM\-generated writing feedback at the granularity instructors work in\.

Our experiments across 74 configurations spanning multiple LLMs show that no setting exceeds0\.40\.4F1\. Models struggle to produce the comments instructors would emphasize\. Feedback verbosity, in the sense of producing more comments rather than longer ones, is the dominant driver of poor alignment\. WhatUniMatchpenalizes is pedagogically meaningful: overly verbose feedback overwhelms the student, leading to counterproductive resultsKluger and DeNisi \([1996](https://arxiv.org/html/2607.00274#bib.bib83)\)\.

## 2Related Work

Dataset nameSize\(\# drafts\)Feedback word countWriting genre\(s\)Total\(corpus\)Inline/draftOverall/draftExposía\(Zyskaet al\.,[2026](https://arxiv.org/html/2607.00274#bib.bib23)\)5517\.6K107213ProposalArgRewrite V\.2\(Kashefiet al\.,[2022](https://arxiv.org/html/2607.00274#bib.bib72)\)8612\.9K–150EssayInsta\-Reviewer\(Jiaet al\.,[2022](https://arxiv.org/html/2607.00274#bib.bib85)\)48426\.6K–55Research ReportSefora\(ours\)564136\.4K14795Essay, Narrative, Explanation, EmpathyTable 1:Comparison of related datasets of student writing with instructor feedback\. The table counts only instructor\-authored feedback \(peer, user, and AI\-augmented excluded\), with per\-draft word counts averaged over each corpus’s total draft count\. Writing genres follow the BAWE classification ofGardner and Nesi \([2013](https://arxiv.org/html/2607.00274#bib.bib16)\)\. Inline denotes span\-anchored feedback; other features \(e\.g\., error tags, scores\) are omitted\.#### Essay datasets\.

Many existing writing datasets target*Automatic Essay Scoring*\(AES\), predicting numerical scores rather than formative feedback\(Ramesh and Sanampudi,[2022](https://arxiv.org/html/2607.00274#bib.bib80); Ke and Ng,[2019](https://arxiv.org/html/2607.00274#bib.bib79); Houet al\.,[2025](https://arxiv.org/html/2607.00274#bib.bib20)\), or emphasize structured annotations \(e\.g\., error tags or analytic scores\) over formative text, including PERSUADE 2\.0\(Crossleyet al\.,[2024](https://arxiv.org/html/2607.00274#bib.bib71)\), ASAP/ASAP\+\+222[https://www\.kaggle\.com/c/asap\-aes](https://www.kaggle.com/c/asap-aes)\(Mathias and Bhattacharyya,[2018](https://arxiv.org/html/2607.00274#bib.bib67)\), and NUCLE\(Dahlmeieret al\.,[2013](https://arxiv.org/html/2607.00274#bib.bib54)\)\. While valuable for large\-scale assessment, scores rate a draft but do not tell a student how to revise it\(Keet al\.,[2019](https://arxiv.org/html/2607.00274#bib.bib78)\); feedback generation instead requires localized, actionable, context\-sensitive comments tied to the writer’s text and the assignment goals\.

Efforts to collect feedback\-based essay datasets have grown in recent years, yet each captures only part of the setting we target\. They pair expert feedback with a narrow slice of writing: Insta\-Reviewer\(Jiaet al\.,[2022](https://arxiv.org/html/2607.00274#bib.bib85)\)with graduate project reports, ArgRewrite V\.2\(Kashefiet al\.,[2022](https://arxiv.org/html/2607.00274#bib.bib72)\)with a single argumentative prompt, and Exposía\(Zyskaet al\.,[2026](https://arxiv.org/html/2607.00274#bib.bib23)\)with research exposés, while CityU\(Leeet al\.,[2015](https://arxiv.org/html/2607.00274#bib.bib56)\)333The original paper described the corpus as not yet publicly available at the time, and we could not verify a current public release\.offers short tutor comments on ESL/EFL writing\. Each is limited on at least one axis central to classroom feedback, lacking multiple drafts or revisions\(Jiaet al\.,[2022](https://arxiv.org/html/2607.00274#bib.bib85); Leeet al\.,[2015](https://arxiv.org/html/2607.00274#bib.bib56)\)or covering only a single genre\(Jiaet al\.,[2022](https://arxiv.org/html/2607.00274#bib.bib85); Kashefiet al\.,[2022](https://arxiv.org/html/2607.00274#bib.bib72); Zyskaet al\.,[2026](https://arxiv.org/html/2607.00274#bib.bib23)\)\. LEAF\(Behzadet al\.,[2024](https://arxiv.org/html/2607.00274#bib.bib22)\)scales to more essay\-feedback pairs but sources them from online users and AI rather than instructors\.Seforacomplements these resources with authentic, instructor\-authored feedback, both span\-anchored and overall, at larger scale across diverse college writing genres \(Table[1](https://arxiv.org/html/2607.00274#S2.T1)\), paired with assignment prompts, rubrics, analytic scores, and a multi\-draft revision structure, enabling analyses that prior corpora do not directly support\.

#### Feedback evaluation\.

Evaluating generated feedback is difficult, as wording and delivery can vary widely even for the same underlying issue\. Reference\-free evaluation raises the issues discussed previously: effective feedback has no single operational definitionPearson \([2022](https://arxiv.org/html/2607.00274#bib.bib8)\); Stahlet al\.\([2024](https://arxiv.org/html/2607.00274#bib.bib9)\), and LLM\-as\-judge proxies inherit this ambiguity while adding biases of their ownZhenget al\.\([2023](https://arxiv.org/html/2607.00274#bib.bib10)\); Deutschet al\.\([2022](https://arxiv.org/html/2607.00274#bib.bib11)\)\. We thus focus on reference\-based and human\-centered evaluation, which fall into three groupsCelikyilmazet al\.\([2020](https://arxiv.org/html/2607.00274#bib.bib68)\)\. Content\-overlap metrics such as BLEUPapineniet al\.\([2002](https://arxiv.org/html/2607.00274#bib.bib65)\)and ROUGELin \([2004](https://arxiv.org/html/2607.00274#bib.bib64)\)reward surface similarity and are brittle for open\-ended feedback\. Model\-based metrics such as BERTScoreZhanget al\.\([2019](https://arxiv.org/html/2607.00274#bib.bib60)\), BARTScoreYuanet al\.\([2021](https://arxiv.org/html/2607.00274#bib.bib61)\), and BLEURTSellamet al\.\([2020](https://arxiv.org/html/2607.00274#bib.bib62)\)capture general semantics better but are hard to interpret and miss whether LLM feedback identifies the same substantive points as the instructor\. Human\-centered evaluation is reliable but costly and hard to scaleJiaet al\.\([2022](https://arxiv.org/html/2607.00274#bib.bib85)\)\.UniMatchtargets both reliability and interpretability by comparing feedback as individual units and scoring their correspondence under feedback\-specific similarity criteria\.

## 3Dataset

![Refer to caption](https://arxiv.org/html/2607.00274v1/x2.png)Figure 2:An annotated draft fromSefora, illustrating the dataset’s span\-anchored, fine\-grained instructor feedback: sticky\-note comments paired with color\-coded highlights\. Pink highlights mark especially effective passages, yellow highlights indicate issues needing attention, and green highlights mark ideas worth developing in a subsequent draft\. Full annotation conventions are documented in §[A\.2](https://arxiv.org/html/2607.00274#A1.SS2)\.Seforawas collected over two semesters from a range of first\-year undergraduate courses in the university’s English Department, spanning composition, ESL, narrative and creative writing, and a senior seminar\. It includes assignment prompts, grading rubrics, multi\-draft student essays, and two forms of instructor feedback: span\-anchored \(inline\) comments \(Figure[2](https://arxiv.org/html/2607.00274#S3.F2)\) and overall assessments\. The dataset is publicly available\.444[https://github\.com/ShayanPey/SEFORA](https://github.com/ShayanPey/SEFORA)

#### Scope, scale, and genres\.

Seforacontains 371 essays by 155 students across 34 assignments from 9 classes, comprising 564 drafts \(220 single\-draft, 109 two\-stage, 42 three\-stage\) that total 8,186 paragraphs\. Under the BAWE classification\(Gardner and Nesi,[2013](https://arxiv.org/html/2607.00274#bib.bib16)\), these span four genre families: Essay, Narrative Recount, Explanation, and Empathy Writing\.555For each assignment we map its writing prompt to the genre family it most closely matches; BAWE’s explicit definitions and examples make this a direct mapping\.Instructors contributed 5,684 inline \(span\-anchored\) annotations – highlights, sticky notes, and strikeouts, nearly all with comment – averaging 10 annotations and 147 feedback words per draft\. Additionally, 437 drafts carry overall comments \(121 words avg\.\), 295 carry analytic trait scores \(6\.5 per draft avg\.\), and 205 a holistic score, for 8,240 instructor annotations in total\.

#### Annotation content\.

Seforapreserves the original paragraph structure and aligns each annotation to the text it targets: most are paragraph\-level and*span\-anchored*– tied to the exact span the instructor marked, whether a phrase, sentence, or word – with document\-level comments and scores retained when present \(details in §[A\.2](https://arxiv.org/html/2607.00274#A1.SS2)\)\. What distinguishesSeforafrom error\-tag and score\-only corpora is the*substance*of its feedback: it is overwhelmingly*facilitative*\(inviting the writer’s own reflection\) rather than*directive*\(prescribing changes\)Straub \([1996](https://arxiv.org/html/2607.00274#bib.bib19)\)\. Rather than flagging surface errors, instructors engage substantive aspects of writing – narrative development, idea elaboration, reader engagement, voice, revision guidance – and frequently pose open questions back to the writer\.

#### Collection, Privacy, and Ethics\.

Data collection was approved by the university’s IRB\. The research team contacted instructors, who emailed students a recruitment notice outlining the study, its objectives, risks, and voluntary participation\. Materials were gathered from instructors and consenting students across two semesters\. All IRB\-related materials are provided in §[A\.1](https://arxiv.org/html/2607.00274#A1.SS1)\. All essays were released with student consent and processed under strict privacy constraints\. We remove direct identifiers \(student and instructor names, course identifiers, submission dates\) and replace names within essay bodies with consistent within\-essay pseudonyms while preserving grammatical and narrative coherence; references to public figures are retained\. We further screen the corpus for indirect identifying information \(e\.g\., social media or sandbox handles\), and check for abusive or harmful content usingomni\-moderation666[https://developers\.openai\.com/api/docs/models/omni\-moderation\-latest](https://developers.openai.com/api/docs/models/omni-moderation-latest)exclusively via the API under OpenAI’s data\-handling policy\.777[https://developers\.openai\.com/api/docs/guides/your\-data/](https://developers.openai.com/api/docs/guides/your-data/)We do not mask vulgar or strong language, as doing so would alter the writer’s voice and distort the feedback context; the dataset is released with appropriate content warnings\.

#### Parsing\.

Original submissions in PDF or\.docxwere converted to a JSON format that preserves both paragraph segmentation and annotation anchoring\. We release the deterministic parser888[https://github\.com/ShayanPey/SEFORA](https://github.com/ShayanPey/SEFORA)to support future work on similar materials\. Full parsing details are in §[A\.3](https://arxiv.org/html/2607.00274#A1.SS3)\.

## 4Evaluation Framework

Beyond a dataset, scaling instructor\-quality feedback requires a way to evaluate it; both reference\-free and reference\-based approaches face challenges\. We introduceUniMatch, a reference\-based framework that compares model and instructor feedback at the level of feedback units rather than as monolithic texts\. Instructor feedback serves as a principled reference for two reasons\. First, it encodes pedagogical priorities – what to flag, what to praise, and how to phrase guidance – per curriculum rubrics and teaching goals\. Second, good writing feedback has no single operational definitionPearson \([2022](https://arxiv.org/html/2607.00274#bib.bib8)\); Stahlet al\.\([2024](https://arxiv.org/html/2607.00274#bib.bib9)\); instructor judgment, while not the only valid target, is the most defensible anchor for a given draft, especially for the open\-ended writing inSeforawhere LLM feedback remains far from human quality\(Chakrabartyet al\.,[2024](https://arxiv.org/html/2607.00274#bib.bib18); Gómez\-Rodríguez and Williams,[2023](https://arxiv.org/html/2607.00274#bib.bib17)\)\.

UniMatchcompares model and instructor feedback in three stages \(Figure[1](https://arxiv.org/html/2607.00274#S1.F1)\): \(i\) segment each into feedback units, \(ii\) score semantic correspondence between instructor\-model unit pairs, and \(iii\) apply maximum bipartite matching over those scores to derive interpretable unit\-level alignments and aggregate precision, recall, and F1\. Stages \(i\) and \(ii\) are independently verifiable against human annotation, enabling robustness checks on the pipeline\.

### 4\.1Feedback Segmentation

Prior work has typically treated feedback as a monolithic message rather than a set of separable unitsWu and Schunn \([2020](https://arxiv.org/html/2607.00274#bib.bib50)\); Lyuet al\.\([2024](https://arxiv.org/html/2607.00274#bib.bib49)\), but holistic comparison cannot tell which individual observations are recovered or missed\. Other work has recognized the importance of feedback units, though with varying definitions: one or more sentences expressing a coherent thought\(Yenet al\.,[2020](https://arxiv.org/html/2607.00274#bib.bib48)\), or a self\-contained message targeting a specific issue\(Zouet al\.,[2024](https://arxiv.org/html/2607.00274#bib.bib47)\)\. Building on these\(Yenet al\.,[2020](https://arxiv.org/html/2607.00274#bib.bib48); Zouet al\.,[2024](https://arxiv.org/html/2607.00274#bib.bib47)\), we adopt*feedback unit*– a self\-contained statement addressing one specific aspect of the student’s writing – as the basic evaluation granularity, and develop an annotation guideline \(§[B\.1](https://arxiv.org/html/2607.00274#A2.SS1)\) for segmenting instructor and LLM\-generated feedback \(examples in Table[2](https://arxiv.org/html/2607.00274#S4.T2)\)\.

This task is related to discourse parsingLiet al\.\([2014](https://arxiv.org/html/2607.00274#bib.bib46)\); Soricut and Marcu \([2003](https://arxiv.org/html/2607.00274#bib.bib44)\); Marcu \([2000](https://arxiv.org/html/2607.00274#bib.bib45)\), but differs in two key ways: we omit inter\-segment relations, and place emphasis on sub\-sentence segmentation, as a single sentence may contain multiple feedback units, while a single unit may span multiple sentences\. We therefore draw on prior work on sub\-sentence segmentation in discourseLiet al\.\([2022](https://arxiv.org/html/2607.00274#bib.bib43)\); Stab and Gurevych \([2017](https://arxiv.org/html/2607.00274#bib.bib40)\); Huaet al\.\([2019](https://arxiv.org/html/2607.00274#bib.bib41)\); Wanget al\.\([2018](https://arxiv.org/html/2607.00274#bib.bib42)\)\.

#### Formalization and agreement metrics\.

LetT=t1,…,tnT=t\_\{1\},\\dots,t\_\{n\}be the sequence of words of a feedback message\. A segmentation is represented by a boundary vectorQ∈\{0,1\}n−1Q\\in\\\{0,1\\\}^\{n\-1\}whereQi=1Q\_\{i\}=1marks a boundary betweentit\_\{i\}andti\+1t\_\{i\+1\}\. We measure segmentation agreement using WindowDiffPevzner and Hearst \([2002](https://arxiv.org/html/2607.00274#bib.bib39)\), a standard discourse segmentation metric, with window sizekkset to half the average reference segment length \(k=10k\{=\}10in our samples\):

W​D​\(r,h\)=1n−k​∑i=1n−k𝟏​\[b​\(ri:i\+k\)≠b​\(hi:i\+k\)\],WD\(r,h\)=\\frac\{1\}\{n\-k\}\\sum\_\{i=1\}^\{n\-k\}\\mathbf\{1\}\\\!\\left\[b\(r\_\{i:i\+k\}\)\\neq b\(h\_\{i:i\+k\}\)\\right\],whererrandhhare the two segmentations being compared, one designated the reference and the other the hypothesis \(the second annotator, or the model when validating automation\), andb​\(xi:j\)b\(x\_\{i:j\}\)is the boundary count betweentit\_\{i\}andtjt\_\{j\}in segmentationxx\. Because WindowDiff tolerates small boundary shifts, we also report boundary precision, recall, and F1calculated over boundary indices while excluding true negatives to avoid inflated agreement\.

#### Automatic segmentation validation\.

Following the segmentation guideline \(§[B\.1](https://arxiv.org/html/2607.00274#A2.SS1)\), two graduate students independently annotated more than 50 feedback instances, containing both instructor and LLM feedback\. Agreement was high: WD =0\.0730\.073, precision =0\.990\.99, recall =0\.940\.94, F1=0\.960\.96\. The annotators then resolved disagreements to produce a consensus segmentation, which we treat as the reference\.

Using the same segmentation guideline given to human annotators as prompt, GPT\-5\-nano reaches comparable agreement against this consensus reference \(WD =0\.0360\.036, precision =1\.001\.00, recall =0\.950\.95, F1=0\.970\.97\), indistinguishable from inter\-annotator agreement\.

\# Segmented feedback1:This is well\-written,but try to make it more concise\.2:Nice\! So much authenticity in your voice\.3:Try to add more sensory details\. Like what was specialabout that night?Also, why did he suddenly leave?Table 2:Examples of feedback segmented into feedback units\.

### 4\.2Feedback Unit Similarity

We then compare the similarity between feedback units\. This task is related to Semantic Textual Similarity \(STS\)Bäret al\.\([2012](https://arxiv.org/html/2607.00274#bib.bib38)\); Majumderet al\.\([2016](https://arxiv.org/html/2607.00274#bib.bib37)\), but differs in two key ways: the texts being compared are feedback units rather than arbitrary sentences, and both address the same paragraph of the same essay\.

We thus introduce a task\-specific guideline \(§[B\.2](https://arxiv.org/html/2607.00274#A2.SS2)\) for scoring feedback unit similarity along two dimensions of feedback: the*target*\(the aspect, span, or idea addressed\) and the*comment*\(the evaluation, critique, or suggestion about that target\)\. Drawing on integer\-scale scoring practice in STSAgirreet al\.\([2012](https://arxiv.org/html/2607.00274#bib.bib34)\); Xuet al\.\([2015](https://arxiv.org/html/2607.00274#bib.bib6)\); Agirreet al\.\([2015](https://arxiv.org/html/2607.00274#bib.bib5)\), we use a 0–4 scale chosen to suit the feedback setting: anchored at 4 \(near\-equivalent as feedback\) and 0 \(irrelevant or contradictory\), with an explicit middle value \(2\) for pairs that share an important point but differ on another \(intermediate levels and details in §[B\.2](https://arxiv.org/html/2607.00274#A2.SS2)\)\. The guideline was developed in collaboration with course instructors and refined iteratively by two graduate student annotators, who piloted it, discussed disagreements, and refined wording and examples until agreement stabilized\. They then independently scored 120 feedback unit pairs spanning the full scale; we report inter\-annotator Pearson and Spearman correlations ofr=0\.7572r=0\.7572andρ=0\.7558\\rho=0\.7558\. We use the mean as the reference similarity for these 120 pairs\.

#### Automatic similarity scorer validation\.

Existing similarity methods fall short on this task: lexical\-overlap metrics yield Pearson correlations below0\.10\.1and embedding\-based methods reach at mostr≈0\.67r\\approx 0\.67, both too low to be reliable \(details in §[C\.1](https://arxiv.org/html/2607.00274#A3.SS1)\)\. We therefore experimented with several LLMs using the defined guideline as prompt, with several exceedingr=0\.7r=0\.7and the strongest exceedingr=0\.8r=0\.8\. However, as scoring every pair independently at our scale would be expensive, we introduce batching \(details in §[C\.2](https://arxiv.org/html/2607.00274#A3.SS2)\), which preserves performance at a fraction of the cost \(Table[3](https://arxiv.org/html/2607.00274#S4.T3)999Model performance can shift over time; results were collected in May 2026\. Result using the snapshot modelgpt\-5\-mini\-2025\-08\-07is also reported for stability\.\)\. We adoptgemini\-3\.1\-flash\-lite\-previewas the similarity scorer in our main pipeline \(r=0\.813r=0\.813andρ=0\.818\\rho=0\.818against the human reference\)\. Figure[3](https://arxiv.org/html/2607.00274#S4.F3)visualizes its agreement on a sample with the pipeline’s natural score distribution\.

ModelVersionSizeReason𝒓\\bm\{r\}𝝆\\bm\{\\rho\}GPT5\.1–low0\.8050\.8050\.7940\.7945†minilow0\.7230\.7230\.7300\.7305nanolow0\.5980\.5980\.6040\.604Gemini3\.1flash\-lite‡minimal0\.8130\.8130\.8180\.8182\.5lite–0\.7010\.7010\.7110\.711Table 3:The Pearsonrrand Spearmanρ\\rhocorrelations between LLM\-generated and human\-annotated similarity scores for paired feedback units \(withbatch\_size=50=50\) over120120samples\.†Snapshotgpt\-5\-mini\-2025\-08\-07\.‡Model namegemini\-3\.1\-flash\-lite\-preview\.annot=4annot=3annot=2annot=1annot=0pred=0pred=1pred=2pred=3pred=4Figure 3:Agreement between annotator\-averaged scores \(rows, rounded to the model’s scale\) andgemini\-3\.1\-flash\-lite\-previewscores \(columns\); darker cells hold more pairs\. Intensity along the diagonal indicates strong agreement; the low\-score corner is darkest as most feedback\-unit pairs are unrelated and score low\.

### 4\.3Feedback Unit Cross Matching

Given pairwise similarity scores between instructor and model\-generated feedback units,UniMatchcomputes a one\-to\-one alignment between the two sets\. Intuitively, each feedback unit is matched to its most similar counterpart on the other side, subject to a globally optimal overall matching \(Figure[4](https://arxiv.org/html/2607.00274#S4.F4)illustrates\)\. Specifically, for instructor feedback unitsG=\{Gi\}i=1\|G\|G=\\\{G\_\{i\}\\\}\_\{i=1\}^\{\|G\|\}and predicted feedback unitsP=\{Pj\}j=1\|P\|P=\\\{P\_\{j\}\\\}\_\{j=1\}^\{\|P\|\}, letϕ​\(Gi,Pj\)≥0\\phi\(G\_\{i\},P\_\{j\}\)\\geq 0denote the similarity scorer\. We define the optimal alignment as the maximum\-weight bipartite matching

M∗=arg⁡maxM∈ℳ​∑\(Gi,Pj\)∈Mϕ​\(Gi,Pj\),M^\{\*\}=\\arg\\max\_\{M\\in\\mathcal\{M\}\}\\sum\_\{\(G\_\{i\},P\_\{j\}\)\\in M\}\\phi\(G\_\{i\},P\_\{j\}\),whereℳ\\mathcal\{M\}is the set of 1\-to\-1 matchings betweenGGandPP\. ComputingM∗M^\{\*\}is a standard maximum\-weight bipartite matching problem, and we use the Hungarian algorithmKuhn \([1955](https://arxiv.org/html/2607.00274#bib.bib24)\)for efficiency\. LetΦ​\(M∗\)=∑\(Gi,Pj\)∈M∗ϕ​\(Gi,Pj\)\\Phi\(M^\{\*\}\)=\\sum\_\{\(G\_\{i\},P\_\{j\}\)\\in M^\{\*\}\}\\phi\(G\_\{i\},P\_\{j\}\)denote the total weight of this alignment\.

To turnΦ​\(M∗\)\\Phi\(M^\{\*\}\)into interpretable precision and recall, we normalize it against per\-side upper bounds, each set’s summed within\-set self\-similarity \(∑jϕ​\(Pj,Pj\)\\sum\_\{j\}\\phi\(P\_\{j\},P\_\{j\}\)predicted,∑iϕ​\(Gi,Gi\)\\sum\_\{i\}\\phi\(G\_\{i\},G\_\{i\}\)instructor\), which equals its unit count times the maximum score \(4\) since each unit matches itself\. Following coreference resolutionLuo \([2005](https://arxiv.org/html/2607.00274#bib.bib26)\), we define soft precision and recall as

p=Φ​\(M∗\)∑jϕ​\(Pj,Pj\),r=Φ​\(M∗\)∑iϕ​\(Gi,Gi\),p=\\frac\{\\Phi\(M^\{\*\}\)\}\{\\sum\_\{j\}\\phi\(P\_\{j\},P\_\{j\}\)\},\\qquad r=\\frac\{\\Phi\(M^\{\*\}\)\}\{\\sum\_\{i\}\\phi\(G\_\{i\},G\_\{i\}\)\},and the soft F1as

F1=2​p​rp\+r\.F\_\{1\}=\\frac\{2pr\}\{p\+r\}\.The result is an interpretable, soft precision/recall view of feedback quality: precision captures how much of the model’s feedback aligns with what an instructor wrote, and recall captures how much of the instructor’s feedback the model recovers\.101010The pipeline also admits a threshold\-based instantiation with hard matches above a chosenτ\\tau, exposing strictness as an explicit knob \(§[D](https://arxiv.org/html/2607.00274#A4)\)\.

A:B:So creative\! Nice\!But try to make it more concise\.This is well\-written,however, it’s a little verbose\.Figure 4:Matching feedbackAandB: boxes mark segments; solid lines = high similarity, dashed = low\.#### End\-to\-end validation\.

UniMatch’s components – segmentation \(§[4\.1](https://arxiv.org/html/2607.00274#S4.SS1)\) and similarity scoring \(§[4\.2](https://arxiv.org/html/2607.00274#S4.SS2)\) – are validated against human judgment in isolation, but this does not guarantee that the composed metric aligns with instructor judgment\. To test the pipeline end to end, we selected 20 paragraphs, each with three raw LLM feedback texts \(§[5](https://arxiv.org/html/2607.00274#S5)\) thatUniMatchscores at well\-separated levels \(F≈10\{\}\_\{1\}\\approx 0,0\.50\.5,11\), and had a writing expert independently rank the three by quality against the reference feedback\.111111Ties were permitted but occurred for only two paragraphs\.Across all 60 texts,UniMatchF1correlates strongly with the instructor ranking \(Pearsonr=0\.836r=0\.836, Spearmanρ=0\.837\\rho=0\.837\), showing that the full pipeline recovers expert judgments\.

## 5Experiments

We evaluate paragraph\-level LLM\-generated feedback underUniMatch, varying model choice and prompting setup to study what task framing and generation choices drive feedback quality\.

#### Task framing and evaluation data\.

We generate feedback at the paragraph level, matching the granularity of inline instructor annotations inSefora\. For each target paragraph, the model receives the paragraph plus as much surrounding context as fits within a fixed 8,192\-token budget, enough to include the entire essay for most drafts\. We evaluate on the 3,499 paragraphs inSeforawith at least one instructor annotation \(*full\-corpus*\); of these, 2,324 carry exactly one \(the*single\-reference*subset\)\. We report metrics on both\.

#### Models\.

To support broad ablations across settings, we primarily use small open\-weight instruction\-tuned models, namely, Llama\-3\.1\-8BGrattafioriet al\.\([2024](https://arxiv.org/html/2607.00274#bib.bib1)\)\(Llama\), DeepSeek\-R1\-Distill\-Llama\-8BGuoet al\.\([2025](https://arxiv.org/html/2607.00274#bib.bib58)\)\(DeepSeek\), and Mistral\-7BJianget al\.\([2023](https://arxiv.org/html/2607.00274#bib.bib57)\)\(Mistral\)\. To probe whether the effects extend to a larger regime, we also evaluate Qwen2\.5\-72B\-InstructQwenet al\.\([2025](https://arxiv.org/html/2607.00274#bib.bib53)\)\(Qwen\) and closed model GPT\-5\.1121212[https://developers\.openai\.com/api/docs/models/gpt\-5\.1](https://developers.openai.com/api/docs/models/gpt-5.1)\(GPT\) on a subset of configurations\.

#### Prompt template\.

We use two prompt templates\. The default*constrained*prompt asks the model to identify the single most important focus in the target paragraph and produce one feedback unit under a prescribed output format; the*unconstrained*prompt places no limit on the number of feedback units it may return\. On top of the template, we vary four further dimensions of task specification: zero\-shot vs\. few\-shot prompting \(5 exemplars\)Brownet al\.\([2020](https://arxiv.org/html/2607.00274#bib.bib4)\); guided vs\. unguided prompting, where the guided setting supplies a list of feedback categoriesNarciss \([2008](https://arxiv.org/html/2607.00274#bib.bib3)\); Keuninget al\.\([2018](https://arxiv.org/html/2607.00274#bib.bib2)\)in the prompt; inclusion vs\. omission of the assignment prompt and rubric; and chain\-of\-thought promptingWeiet al\.\([2022](https://arxiv.org/html/2607.00274#bib.bib52)\); Lyuet al\.\([2023](https://arxiv.org/html/2607.00274#bib.bib51)\)\. The guided categories cover the common functional roles of writing feedback:*Task Constraints*,*Concepts*,*Elaboration*,*Clarification*,*Mistakes*, and*Praise*\. Few\-shot exemplars are 5 instructor\-authored feedback samples covering each category\. Full prompt templates and guided variants are in §[F](https://arxiv.org/html/2607.00274#A6)\.

#### Generation settings\.

We allocate up to 250 output tokens per target paragraph – well above the typical length of instructor feedback on a single paragraph inSefora\(≈20\\approx 20tokens131313Exact number of tokens depends on the tokenizer\.\) – to avoid truncating multi\-point responses\.

In total, the model and prompt combination yields 74 experimental configurations\. All prompts \(§[F](https://arxiv.org/html/2607.00274#A6)\), inference pipelines, theUniMatchframework, and code are publicly available\.141414[https://github\.com/ShayanPey/SEFORA](https://github.com/ShayanPey/SEFORA)

## 6Results

ModelPrecisionRecallF1Ablations \(F1\)Single\-referenceUnconstrainedUnguidedNo rubricsFew\-shotLlama\_8B0\.2750\.2360\.2540\.2900\.1130\.2030\.2680\.271Mistral\_7B0\.2880\.2670\.2770\.3110\.1590\.2510\.2780\.258Qwen\_72B0\.3740\.2680\.3120\.3870\.1790\.2710\.2850\.317GPT\-5\.10\.3760\.2730\.3160\.3710\.1360\.2960\.2880\.307Table 4:UniMatchresults for the configuration <zero\-shot, rubric, guided, default prompt\> on the full corpus\. Left: precision, recall, F1\. Right: F1under single\-axis variations –*Single\-reference*subset \(paragraphs with exactly one instructor annotation\),*Unconstrained*prompt,*Unguided*,*No rubrics*, and*Few\-shot*\. Full per\-configuration results in Tables[5](https://arxiv.org/html/2607.00274#A5.T5)–[7](https://arxiv.org/html/2607.00274#A5.T7)\(§[E](https://arxiv.org/html/2607.00274#A5)\)\.Table[4](https://arxiv.org/html/2607.00274#S6.T4)reportsUniMatchscores on the full corpus under the zero\-shot setting \(with rubrics and category guidance\), where most models perform best; full results are in Tables[5](https://arxiv.org/html/2607.00274#A5.T5)–[7](https://arxiv.org/html/2607.00274#A5.T7)of §[E](https://arxiv.org/html/2607.00274#A5)\. No configuration exceeds0\.40\.4F1, and both precision and recall stay low: even the strongest GPT model reaches only0\.380\.38precision and0\.270\.27recall, so models miss most of the feedback instructors prioritize while spending much of their output on points instructors do not raise\. Among the small models \(≤8\\leq 8B parameters\), Mistral is consistently strongest, ahead of Llama151515Sometimes DeepSeek as shown in §[E](https://arxiv.org/html/2607.00274#A5)\.\. The larger Qwen and GPT lead overall – GPT on the shared subset, Qwen on the single best configuration \(0\.320\.32F1\) – but only narrowly, and all remain far from matching instructor feedback\.

Table[4](https://arxiv.org/html/2607.00274#S6.T4)additionally presents ablations: each varies a single variable relative to the base setting <zero\-shot, rubric, guided, default prompt\>\. We show a subset due to space constraints, but the pattern matches the full results\. On the single\-reference subset \(fourth column\), scores are higher as expected, since our constrained default prompt emits one feedback unit per paragraph and thus naturally aligns more with single\-reference paragraphs\.

#### Verbosity\.

4k5k7k12k19k0\.10\.10\.20\.20\.30\.30\.40\.4PromptNumber of generated feedback unitsScorePrecisionRecallF1Figure 5:Precision, recall, and F1across the 74 configurations of §[5](https://arxiv.org/html/2607.00274#S5), sorted by number of generated feedback units \(xx\-axis is ordinal and not to scale\)\. The strip marks the prompt template: the constrained template \(dark\) and the unconstrained \(light\)\. Precision and F1fall as volume rises while recall does not compensate\.The dominant signal in our experiments is verbosity\.Across all models, the constrained prompt \(our default\) beats the unconstrained alternative, which yields substantially lower F1\(the*unconstrained*column of Table[4](https://arxiv.org/html/2607.00274#S6.T4)\)\. Removing the single\-feedback constraint lets models emit more than one feedback unit per paragraph, and the resulting over\-generation sharply lowers precision\. This holds even on the full corpus, where paragraphs often carry multiple instructor units and the unconstrained prompt should in principle have the edge: its extra units add low\-similarity matches that depress precision more than they lift recall\. Figure[5](https://arxiv.org/html/2607.00274#S6.F5)confirms the pattern across all evaluated configurations – the number of generated feedback units is strongly negatively correlated with both precision and F1\(r=−0\.850r=\-0\.850andr=−0\.840r=\-0\.840, respectively\), while the accompanying recall gains are too modest to compensate \(full results in §[E](https://arxiv.org/html/2607.00274#A5)\)\.

#### Guiding the model with feedback categories\.

Supplying explicit feedback categories \(the*guided*setting\) is the next strongest factor: it improves F1by about\+0\.059\+0\.059without rubric and\+0\.035\+0\.035with rubric on the full\-corpus evaluation in matched comparisons \(the*Unguided*column of Table[4](https://arxiv.org/html/2607.00274#S6.T4); per\-configuration numbers in Tables[5](https://arxiv.org/html/2607.00274#A5.T5)–[7](https://arxiv.org/html/2607.00274#A5.T7), and a per\-model breakdown in Figure[14](https://arxiv.org/html/2607.00274#A5.F14), §[E](https://arxiv.org/html/2607.00274#A5)\)\.

#### Rubric and few\-shot prompting\.

Two further levers contribute little \(Table[4](https://arxiv.org/html/2607.00274#S6.T4),*No rubrics*and*Few\-shot*columns\)\. Rubric inclusion is essentially neutral, with small, unstable differences: in our paragraph\-level setting well\-framed prompts already supply most of the structure a rubric would contribute, and rubrics largely encode whether a draft is on\-task – which, in authentic coursework, it usually is, leaving little for a rubric to anchor given thatSefora’s feedback is facilitative \(§[3](https://arxiv.org/html/2607.00274#S3)\), engaging execution rather than rubric conformance\. Few\-shot prompting is beneficial but secondary, improving F1by about\+0\.017\+0\.017on the full corpus, mainly through precision; its gains are small under the default prompt, as exemplars help most when task framing is weak, and once the prompt already induces the desired behavior, demonstrations add little\. Larger few\-shot gains under the unconstrained alternative are in Table[6](https://arxiv.org/html/2607.00274#A5.T6)\(§[E](https://arxiv.org/html/2607.00274#A5)\); a small chain\-of\-thought study on the three small models follows the same pattern \(§[E](https://arxiv.org/html/2607.00274#A5)\)\. Per\-model breakdowns of both effects appear in Figures[15\(a\)](https://arxiv.org/html/2607.00274#A5.F15.sf1)and[15\(b\)](https://arxiv.org/html/2607.00274#A5.F15.sf2)\(§[E](https://arxiv.org/html/2607.00274#A5)\)\.

## 7Conclusion

We introduceSefora, a corpus of authentic instructor feedback on student writing, andUniMatch, a reference\-based framework for evaluating generated feedback\. Across 74 configurations, no setting exceeds0\.40\.4F1: the central challenge for LLMs is not producing feedback but producing the comments an instructor would prioritize\. Over\-generation is consistently costly, with precision dropping sharply as models generate more while recall rarely compensates, a pattern that is pedagogically meaningful, since overwhelming a student with feedback is itself counterproductiveKluger and DeNisi \([1996](https://arxiv.org/html/2607.00274#bib.bib83)\)\. Validated end\-to\-end against expert judgment,UniMatchprovides a reliable automatic metric for this task, and together withSeforalays a foundation for future work on instructor\-aligned feedback generation across diverse college writing genres\.

## 8Limitations

AlthoughUniMatchenables fine\-grained analysis, its evaluation operates on feedback units as segmented, without further decomposing compound statements\. Some units express multiple claims \(e\.g\., ‘this paragraph is detailed and smooth’\) and could in principle be split further\. Future work could incorporate automatic claim decomposition to isolate individual feedback propositions at the atomic level\.

Our evaluation operates at the paragraph level: the model is given a target paragraph, andUniMatchassesses the feedback it produces rather than whether the model identifies which span warrants comment in the first place\.Sefora’s span\-anchored annotations make span identification directly studiable, but we leave this setting to future work\. Our experiments also evaluate against the inline \(span\-anchored\) feedback only, leaving the document\-level overall assessments and analytic scores inSeforafor future work\.

Our alignment uses maximum\-weight bipartite matching over all eligible unit pairs\. Alternative alignment constraints \(e\.g\., prioritizing globally highest\-scoring pairs before enforcing one\-to\-one matching\) may produce different match structures and could be explored as alternative objectives\.

Our pipeline relies on closed\-source models \(GPT\-5\-nano for segmentation andgemini\-3\.1\-flash\-lite\-previewfor similarity scoring\), chosen because they meet the human\-agreement bar required for reliable evaluation in our setting\. This reliance has practical downsides: hosted models can change or be deprecated over time, and their use introduces cost and reproducibility constraints\. A natural next step is to distill or fine\-tune smaller open\-source models for both the segmentation and similarity stages, using our human\-annotated subsets as supervision\. Because each stage is independently validated against human annotation \(§[4\.1](https://arxiv.org/html/2607.00274#S4.SS1), §[4\.2](https://arxiv.org/html/2607.00274#S4.SS2)\), open\-source replacements can be substituted into the pipeline whenever they reach comparable agreement, without re\-validatingUniMatchas a whole\.

UniMatchtreats instructor feedback as ground truth\. Reference\-free evaluation may also be possible by leveraging methods from the automated essay scoring \(AES\) literature\. For example, generated feedback could be applied to the relevant essay span and the resulting revision evaluated with a quality scorer to estimate feedback effectiveness\. Developing such metrics is beyond the scope of this work\.

Finally, our study relies primarily on automatic evaluation under the proposed framework and does not include a complementary human evaluation of overall feedback quality\. WhileUniMatchis designed to better reflect semantic, point\-level alignment with instructor feedback than whole\-text overlap metrics, it does not directly assess dimensions such as helpfulness, pedagogical appropriateness, or actionability as perceived by human readers\. A targeted human study, or an LLM\-as\-judge evaluation calibrated against human judgments, would provide a useful complementary perspective\. We leave such validation to future work\.

## References

- E\. Agirre, C\. Banea, C\. Cardie, D\. Cer, M\. Diab, A\. Gonzalez\-Agirre, W\. Guo, I\. Lopez\-Gazpio, M\. Maritxalar, R\. Mihalcea,et al\.\(2015\)Semeval\-2015 task 2: semantic textual similarity, english, spanish and pilot on interpretability\.InProceedings of the 9th international workshop on semantic evaluation \(SemEval 2015\),pp\. 252–263\.Cited by:[§4\.2](https://arxiv.org/html/2607.00274#S4.SS2.p2.2)\.
- Semeval\-2012 task 6: a pilot on semantic textual similarity\. in\* sem 2012: the first joint conference on lexical and computational semantics–volume 1: proceedings of the main conference and the shared task, and volume 2: proceedings of the sixth international workshop on semantic evaluation \(semeval 2012\)\.InProceedings of the Sixth International Workshop on Semantic Evaluation \(SemEval 2012\), Montréal, QC, Canada,pp\. 7–8\.Cited by:[§4\.2](https://arxiv.org/html/2607.00274#S4.SS2.p2.2)\.
- M\. M\. Ahea, M\. R\. K\. Ahea, and I\. Rahman \(2016\)The value and effectiveness of feedback in improving students’ learning and professionalizing teaching in higher education\.\.Journal of Education and Practice7\(16\),pp\. 38–41\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p1.1)\.
- A\. N\. Applebee and J\. A\. Langer \(2011\)EJ extra: a snapshot of writing instruction in middle schools and high schools \[free access\]\.English journal100\(6\),pp\. 14–27\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p1.1)\.
- S\. K\. Banihashem, N\. T\. Kerman, O\. Noroozi, J\. Moon, and H\. Drachsler \(2024\)Feedback sources in essay writing: peer\-generated or ai\-generated feedback?\.International Journal of Educational Technology in Higher Education21\(1\),pp\. 23\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p1.1)\.
- D\. Bär, C\. Biemann, I\. Gurevych, and T\. Zesch \(2012\)Ukp: computing semantic textual similarity by combining multiple content similarity measures\.In\* SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation \(SemEval 2012\),pp\. 435–440\.Cited by:[§4\.2](https://arxiv.org/html/2607.00274#S4.SS2.p1.1)\.
- S\. Behzad, O\. Kashefi, and S\. Somasundaran \(2024\)LEAF: language learners’ english essays and feedback corpus\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),pp\. 433–442\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p2.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p2.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§5](https://arxiv.org/html/2607.00274#S5.SS0.SSS0.Px3.p1.1)\.
- D\. Carless and D\. Boud \(2018\)The development of student feedback literacy: enabling uptake of feedback\.Assessment & Evaluation in Higher Education43\(8\),pp\. 1315–1325\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p1.1)\.
- A\. Celikyilmaz, E\. Clark, and J\. Gao \(2020\)Evaluation of text generation: a survey\.arXiv preprint arXiv:2006\.14799\.Cited by:[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Chakrabarty, P\. Laban, D\. Agarwal, S\. Muresan, and C\. Wu \(2024\)Art or artifice? large language models and the false promise of creativity\.InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems,pp\. 1–34\.Cited by:[§4](https://arxiv.org/html/2607.00274#S4.p1.1)\.
- S\. A\. Crossley, Y\. Tian, P\. Baffour, A\. Franklin, M\. Benner, and U\. Boser \(2024\)A large\-scale corpus for assessing written argumentation: persuade 2\.0\.Assessing Writing61,pp\. 100865\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p2.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Dahlmeier, H\. T\. Ng, and S\. M\. Wu \(2013\)Building a large annotated corpus of learner english: the nus corpus of learner english\.InProceedings of the eighth workshop on innovative use of NLP for building educational applications,pp\. 22–31\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p2.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Deutsch, R\. Dror, and D\. Roth \(2022\)On the limitations of reference\-free evaluations of generated text\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Abu Dhabi, United Arab Emirates\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p3.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Endignoux, O\. Levillain, and J\. Migeon \(2016\)Caradoc: a pragmatic approach to pdf parsing and validation\.In2016 IEEE Security and Privacy Workshops \(SPW\),pp\. 126–139\.Cited by:[§A\.3](https://arxiv.org/html/2607.00274#A1.SS3.p1.1)\.
- M\. Everingham, L\. Van Gool, C\. K\. I\. Williams, J\. Winn, and A\. Zisserman \(2010\)The PASCAL Visual Object Classes \(VOC\) Challenge\.International Journal of Computer Vision88\(2\),pp\. 303–338\.Cited by:[Appendix D](https://arxiv.org/html/2607.00274#A4.p1.3)\.
- J\. Fernandez, C\. Na, V\. Tiwari, Y\. Bisk, S\. Luccioni, and E\. Strubell \(2025\)Energy considerations of large language model inference and efficiency optimizations\.arXiv preprint arXiv:2504\.17674\.Cited by:[§A\.3](https://arxiv.org/html/2607.00274#A1.SS3.p2.1)\.
- S\. Gardner and H\. Nesi \(2013\)A classification of genre families in university student writing\.Applied linguistics34\(1\),pp\. 25–52\.Cited by:[Table 1](https://arxiv.org/html/2607.00274#S2.T1),[§3](https://arxiv.org/html/2607.00274#S3.SS0.SSS0.Px1.p1.1)\.
- C\. Gómez\-Rodríguez and P\. Williams \(2023\)A confederacy of models: a comprehensive evaluation of llms on creative writing\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 14504–14528\.Cited by:[§4](https://arxiv.org/html/2607.00274#S4.p1.1)\.
- S\. Graham \(2019\)Changing how writing is taught\.Review of Research in Education43\(1\),pp\. 277–303\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§5](https://arxiv.org/html/2607.00274#S5.SS0.SSS0.Px2.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§5](https://arxiv.org/html/2607.00274#S5.SS0.SSS0.Px2.p1.1)\.
- J\. Hattie and H\. Timperley \(2007\)The power of feedback\.Review of educational research77\(1\),pp\. 81–112\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p1.1)\.
- Z\. J\. Hou, A\. Ciuba, and X\. L\. Li \(2025\)Improve llm\-based automatic essay scoring with linguistic features\.arXiv preprint arXiv:2502\.09497\.Cited by:[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Hua, M\. Nikolov, N\. Badugu, and L\. Wang \(2019\)Argument mining for understanding peer reviews\.arXiv preprint arXiv:1903\.10104\.Cited by:[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p2.1)\.
- L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin,et al\.\(2025\)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1–55\.Cited by:[§A\.3](https://arxiv.org/html/2607.00274#A1.SS3.p2.1)\.
- Q\. Jia, M\. Young, Y\. Xiao, J\. Cui, C\. Liu, P\. Rashid, and E\. Gehringer \(2022\)Insta\-reviewer: a data\-driven approach for generating instant feedback on students’ project reports\.\.International Educational Data Mining Society\.Cited by:[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2607.00274#S2.T1.1.5.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§5](https://arxiv.org/html/2607.00274#S5.SS0.SSS0.Px2.p1.1)\.
- O\. Kashefi, T\. Afrin, M\. Dale, C\. Olshefski, A\. Godley, D\. Litman, and R\. Hwa \(2022\)Argrewrite v\. 2: an annotated argumentative revisions corpus\.Language Resources and Evaluation56\(3\),pp\. 881–915\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p2.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p2.1),[Table 1](https://arxiv.org/html/2607.00274#S2.T1.1.4.1)\.
- Z\. Ke, H\. Inamdar, H\. Lin, and V\. Ng \(2019\)Give me more feedback ii: annotating thesis strength and related attributes in student essays\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 3994–4004\.Cited by:[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Ke and V\. Ng \(2019\)Automated essay scoring: a survey of the state of the art\.\.InIJCAI,Vol\.19,pp\. 6300–6308\.Cited by:[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Keuning, J\. Jeuring, and B\. Heeren \(2018\)A systematic literature review of automated feedback generation for programming exercises\.ACM Transactions on Computing Education \(TOCE\)19\(1\),pp\. 1–43\.Cited by:[§5](https://arxiv.org/html/2607.00274#S5.SS0.SSS0.Px3.p1.1)\.
- A\. N\. Kluger and A\. DeNisi \(1996\)The effects of feedback interventions on performance: a historical review, a meta\-analysis, and a preliminary feedback intervention theory\.\.Psychological bulletin119\(2\),pp\. 254\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p1.1),[§1](https://arxiv.org/html/2607.00274#S1.p5.2),[§7](https://arxiv.org/html/2607.00274#S7.p1.2)\.
- H\. W\. Kuhn \(1955\)The hungarian method for the assignment problem\.Naval research logistics quarterly2\(1\-2\),pp\. 83–97\.Cited by:[§4\.3](https://arxiv.org/html/2607.00274#S4.SS3.p1.8)\.
- J\. Lee, C\. Y\. Yeung, A\. Zeldes, M\. Reznicek, A\. Lüdeling, and J\. Webster \(2015\)CityU corpus of essay drafts of english language learners: a corpus of textual revision in second language writing\.Language Resources and Evaluation49\(3\),pp\. 659–683\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p2.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p2.1)\.
- J\. Li, M\. Liu, B\. Qin, and T\. Liu \(2022\)A survey of discourse parsing\.Frontiers of Computer Science16\(5\),pp\. 165329\.Cited by:[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p2.1)\.
- J\. Li, R\. Li, and E\. Hovy \(2014\)Recursive deep models for discourse parsing\.InProceedings of the 2014 conference on empirical methods in natural language processing \(EMNLP\),pp\. 2061–2069\.Cited by:[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p2.1)\.
- C\. Lin \(2004\)Rouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1)\.
- A\. A\. Lipnevich and J\. K\. Smith \(2009\)“I really need feedback to learn:” students’ perspectives on the effectiveness of the differential feedback messages\.Educational Assessment, Evaluation and Accountability21\(4\),pp\. 347–367\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p1.1)\.
- C\. Liu, R\. Lowe, I\. V\. Serban, M\. Noseworthy, L\. Charlin, and J\. Pineau \(2016\)How not to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 2122–2132\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p3.1)\.
- X\. Luo \(2005\)On coreference resolution performance metrics\.InProceedings of human language technology conference and conference on empirical methods in natural language processing,pp\. 25–32\.Cited by:[§4\.3](https://arxiv.org/html/2607.00274#S4.SS3.p2.3)\.
- Q\. Lyu, W\. Chen, J\. Su, and K\. H\. J\. G\. Heng \(2024\)Steps to implementation: the role of peer feedback inner structure on feedback implementation\.Assessment & Evaluation in Higher Education49\(4\),pp\. 572–585\.Cited by:[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p1.1)\.
- Q\. Lyu, S\. Havaldar, A\. Stein, L\. Zhang, D\. Rao, E\. Wong, M\. Apidianaki, and C\. Callison\-Burch \(2023\)Faithful chain\-of\-thought reasoning\.InThe 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics \(IJCNLP\-AACL 2023\),Cited by:[§5](https://arxiv.org/html/2607.00274#S5.SS0.SSS0.Px3.p1.1)\.
- G\. Majumder, P\. Pakray, A\. Gelbukh, and D\. Pinto \(2016\)Semantic textual similarity methods, tools, and applications: a survey\.Computación y Sistemas20\(4\),pp\. 647–665\.Cited by:[§4\.2](https://arxiv.org/html/2607.00274#S4.SS2.p1.1)\.
- D\. Marcu \(2000\)The theory and practice of discourse parsing and summarization\.MIT press\.Cited by:[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p2.1)\.
- S\. Mathias and P\. Bhattacharyya \(2018\)ASAP\+\+: enriching the asap automated essay grading dataset with essay attribute scores\.InProceedings of the eleventh international conference on language resources and evaluation \(LREC 2018\),Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p2.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Narciss \(2008\)Feedback strategies for interactive learning tasks\.InHandbook of research on educational communications and technology,pp\. 125–143\.Cited by:[§5](https://arxiv.org/html/2607.00274#S5.SS0.SSS0.Px3.p1.1)\.
- J\. Novikova, O\. Dušek, A\. C\. Curry, and V\. Rieser \(2017\)Why we need new evaluation metrics for nlg\.InProceedings of the 2017 conference on empirical methods in natural language processing,pp\. 2241–2252\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p3.1)\.
- K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu \(2002\)Bleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§C\.1](https://arxiv.org/html/2607.00274#A3.SS1.p1.4),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1)\.
- W\. S\. Pearson \(2022\)A typology of the characteristics of teachers’ written feedback comments on second language writing\.Cogent Education9\(1\),pp\. 2024937\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p3.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2607.00274#S4.p1.1)\.
- L\. Pevzner and M\. A\. Hearst \(2002\)A critique and improvement of an evaluation metric for text segmentation\.Computational Linguistics28\(1\),pp\. 19–36\.Cited by:[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.SSS0.Px1.p1.7)\.
- M\. Popović \(2015\)ChrF: character n\-gram f\-score for automatic mt evaluation\.InProceedings of the tenth workshop on statistical machine translation,pp\. 392–395\.Cited by:[§C\.1](https://arxiv.org/html/2607.00274#A3.SS1.p1.4)\.
- Qwen, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§5](https://arxiv.org/html/2607.00274#S5.SS0.SSS0.Px2.p1.1)\.
- D\. Ramesh and S\. K\. Sanampudi \(2022\)An automated essay scoring systems: a systematic literature review\.Artificial Intelligence Review55\(3\),pp\. 2495–2527\.Cited by:[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Reiter \(2018\)A structured review of the validity of bleu\.Computational Linguistics44\(3\),pp\. 393–401\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p3.1)\.
- S\. Samsi, D\. Zhao, J\. McDonald, B\. Li, A\. Michaleas, M\. Jones, W\. Bergeron, J\. Kepner, D\. Tiwari, and V\. Gadepally \(2023\)From words to watts: benchmarking the energy costs of large language model inference\.In2023 IEEE High Performance Extreme Computing Conference \(HPEC\),pp\. 1–9\.Cited by:[§A\.3](https://arxiv.org/html/2607.00274#A1.SS3.p2.1)\.
- T\. Sellam, D\. Das, and A\. Parikh \(2020\)BLEURT: learning robust metrics for text generation\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 7881–7892\.Cited by:[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Song, X\. Tan, T\. Qin, J\. Lu, and T\. Liu \(2020\)Mpnet: masked and permuted pre\-training for language understanding\.Advances in neural information processing systems33,pp\. 16857–16867\.Cited by:[§C\.1](https://arxiv.org/html/2607.00274#A3.SS1.p1.4)\.
- R\. Soricut and D\. Marcu \(2003\)Sentence level discourse parsing using syntactic and lexical information\.InProceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics,pp\. 228–235\.Cited by:[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p2.1)\.
- C\. Stab and I\. Gurevych \(2017\)Parsing argumentation structures in persuasive essays\.Computational Linguistics43\(3\),pp\. 619–659\.Cited by:[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p2.1)\.
- M\. Stahl, L\. Biermann, A\. Nehring, and H\. Wachsmuth \(2024\)Exploring llm prompting strategies for joint essay scoring and feedback generation\.InProceedings of the 19th workshop on innovative use of NLP for building educational applications \(BEA 2024\),pp\. 283–298\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p3.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2607.00274#S4.p1.1)\.
- R\. Straub \(1996\)The concept of control in teacher response: defining the varieties of “directive” and “facilitative” commentary\.College Composition & Communication47\(2\),pp\. 223–251\.Cited by:[§3](https://arxiv.org/html/2607.00274#S3.SS0.SSS0.Px2.p1.1)\.
- W\. Wang, F\. Wei, L\. Dong, H\. Bao, N\. Yang, and M\. Zhou \(2020\)Minilm: deep self\-attention distillation for task\-agnostic compression of pre\-trained transformers\.Advances in neural information processing systems33,pp\. 5776–5788\.Cited by:[§C\.1](https://arxiv.org/html/2607.00274#A3.SS1.p1.4)\.
- Y\. Wang, S\. Li, and J\. Yang \(2018\)Toward fast and accurate neural discourse segmentation\.arXiv preprint arXiv:1808\.09147\.Cited by:[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p2.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§5](https://arxiv.org/html/2607.00274#S5.SS0.SSS0.Px3.p1.1)\.
- Y\. Wu and C\. D\. Schunn \(2020\)From feedback to revisions: effects of feedback features and perceptions\.Contemporary Educational Psychology60,pp\. 101826\.Cited by:[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p1.1)\.
- W\. Xu, C\. Callison\-Burch, and W\. B\. Dolan \(2015\)Semeval\-2015 task 1: paraphrase and semantic similarity in twitter \(pit\)\.InProceedings of the 9th international workshop on semantic evaluation \(SemEval 2015\),pp\. 1–11\.Cited by:[§4\.2](https://arxiv.org/html/2607.00274#S4.SS2.p2.2)\.
- Y\. G\. Yen, J\. O\. Kim, and B\. P\. Bailey \(2020\)Decipher: an interactive visualization tool for interpreting unstructured design feedback from multiple providers\.InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems,pp\. 1–13\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p3.1),[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p1.1)\.
- W\. Yuan, G\. Neubig, and P\. Liu \(2021\)Bartscore: evaluating generated text as text generation\.Advances in neural information processing systems34,pp\. 27263–27277\.Cited by:[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi \(2019\)Bertscore: evaluating text generation with bert\.arXiv preprint arXiv:1904\.09675\.Cited by:[§C\.1](https://arxiv.org/html/2607.00274#A3.SS1.p1.4),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\), Datasets and Benchmarks Track,Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p3.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Zou, K\. Guo, J\. Wang, and Y\. Liu \(2024\)Investigating students’ uptake of teacher\-and chatgpt\-generated feedback in efl writing: a comparison study\.Computer Assisted Language Learning,pp\. 1–30\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p3.1),[§4\.1](https://arxiv.org/html/2607.00274#S4.SS1.p1.1)\.
- D\. Zyska, A\. Rozovskaya, I\. Kuznetsov, and I\. Gurevych \(2026\)Exposía: academic writing assessment of exposés and peer feedback\.arXiv preprint arXiv:2601\.06536\.Cited by:[§1](https://arxiv.org/html/2607.00274#S1.p2.1),[§2](https://arxiv.org/html/2607.00274#S2.SS0.SSS0.Px1.p2.1),[Table 1](https://arxiv.org/html/2607.00274#S2.T1.1.3.1)\.

## Appendix ADataset

### A\.1Data Collection Materials

The study was approved by the university’s IRB \(§[3](https://arxiv.org/html/2607.00274#S3.SS0.SSS0.Px3)\)\. The recruitment materials distributed to students and instructors are reproduced below\.

![Refer to caption](https://arxiv.org/html/2607.00274v1/x3.png)Figure 6:Recruitment email for students\.![Refer to caption](https://arxiv.org/html/2607.00274v1/x4.png)Figure 7:Recruitment email for instructor part 1\.![Refer to caption](https://arxiv.org/html/2607.00274v1/x5.png)Figure 8:Recruitment email for instructor part 2\.
### A\.2Annotation Representation Details

Paragraph\-level annotations are stored within the corresponding paragraph, while document\-level comments and scores are stored outside the paragraph structure\.

Sticky notes are stored with their text and exact positions within the paragraph body, reflecting the original placement\. Highlights are stored with their exact spans and associated comments; in our data, nearly all highlights contain instructor\-written comment text\. For all paragraph\-level annotations we also retain the immediate left and right context of the marked span to facilitate locating the annotation within the paragraph, especially when the highlighted text is short or when a sticky note is tied to a specific local position in the essay\. When instructors use non\-generic highlight colors, we preserve these as structured signals\. In this dataset, color usage follows a consistent instructor\-defined legend:

- •Pink: “great job, this is an awesome line”
- •Green: “there is something really interesting here …expand on this point in your next draft”
- •Yellow: “there is some problem with this sentence”

### A\.3Parsing Details

A central preprocessing challenge is preserving both paragraph structure and annotation anchoring across annotated PDF and\.docxsubmissions\. For\.docx, we extract standard highlights and comments with their anchored spans\. For PDFs, however, annotations must be reconstructed from page\-level geometry, including highlight quadrilaterals and note positions, rather than from a linear text streamEndignouxet al\.\([2016](https://arxiv.org/html/2607.00274#bib.bib76)\)\.

Naive PDF text extraction can interleave headers and footers and scramble paragraph boundaries, which are often conveyed by layout rather than explicit markup\. We also explored LLM\-based re\-paragraphization of extracted text, but observed deviations from the source text, including missing or added tokens and altered boundaries, consistent with known hallucination behaviorHuanget al\.\([2025](https://arxiv.org/html/2607.00274#bib.bib75)\); it also incurs nontrivial computational costFernandezet al\.\([2025](https://arxiv.org/html/2607.00274#bib.bib74)\); Samsiet al\.\([2023](https://arxiv.org/html/2607.00274#bib.bib73)\)\. We therefore implemented a deterministic parser that converts annotated PDF and\.docxsubmissions into a unified JSON format while preserving paragraph segmentation and annotation anchoring\. We release this parser with the dataset\. The remainder of this section describes the PDF pipeline, which is the more involved of the two; the stages are summarized in Figure[9](https://arxiv.org/html/2607.00274#A1.F9)\.

#### Body text extraction\.

The parser first extracts the essay body from the page\-level text blocks\. This stage is fully deterministic: every character present in the source text blocks is retained, and no content is dropped, rewritten, or reordered by a model\. The extracted body text serves as the canonical reference string to which all subsequent annotations are anchored\.

#### Highlight anchoring\.

Highlights are recovered from their page\-level geometry rather than from a linear text stream\. For each highlight, the parser collects the highlight quadrilaterals, determines the text covered by those quads, and matches the covered text back to the corresponding span in the extracted body, anchoring the highlight to that span\. The associated highlight comment, when present, is stored alongside the anchored span\. Because nearly all highlights inSeforacarry instructor\-written comment text \(§[A\.2](https://arxiv.org/html/2607.00274#A1.SS2)\), this step preserves both the marked span and its commentary\.

#### Highlight color recovery\.

Highlight color is not reliably available as a structured attribute, so the parser recovers it visually: it performs a localized rendering of the highlighted region, reads the rendered color, and maps the resulting HEX value to the nearest entry in our set of named highlight colors\. This yields the color names used as structured signals in the corpus, following the instructor\-defined legend documented in §[A\.2](https://arxiv.org/html/2607.00274#A1.SS2)\(e\.g\., pink, green, and yellow\), as illustrated in Figure[2](https://arxiv.org/html/2607.00274#S3.F2)\.

#### Sticky\-note anchoring\.

Sticky notes are positioned by page coordinates rather than attached to a text span, so the parser anchors them by location\. For each note, it identifies the surrounding text within a coordinate radius of the note’s placement, locates that surrounding text within the extracted body, and injects the note at the corresponding position\. This reproduces the original placement of the note within the paragraph body \(§[A\.2](https://arxiv.org/html/2607.00274#A1.SS2)\)\.

#### Paragraph structure\.

Text blocks do not necessarily correspond to paragraphs, so paragraph boundaries cannot be read directly from the block structure\. Instead, the parser infers paragraph structure from visual layout cues, primarily the spacing between text blocks, recovering the paragraph segmentation that layout conveys without explicit markup\.

#### Header, footer, and identifier removal\.

Recurring page elements such as top\-of\-page names and page numbers are removed based on their position on the page\. Submission metadata such as the submission date is detected using regular\-expression patterns\. To ensure that student and instructor names do not leak into the released corpus, positional and regex\-based removal was complemented by additional scanning tools and human inspection, consistent with the privacy procedures described in §[3](https://arxiv.org/html/2607.00274#S3.SS0.SSS0.Px3)\.

1\. Body text extractionDeterministic\. Extracts every character in the box\.2\. Highlight anchoringQuads→\\rightarrowcovered text→\\rightarrowmatched span; comment stored\.3\. Color recoveryLocal render→\\rightarrowHEX→\\rightarrownamed color\.4\. Sticky\-note anchoringCoordinate\-radius context→\\rightarrowlocated in body→\\rightarrowinjected\.5\. Paragraph structureInferred from inter\-block spacing \(visual cues\)\.6\. Identifier removalPositional headers/footers; regex metadata; human\-checked\.Figure 9:Stages of the deterministic PDF parsing pipeline \(§[A\.3](https://arxiv.org/html/2607.00274#A1.SS3)\)\. Each annotated submission is converted to a unified JSON representation preserving paragraph segmentation and annotation anchoring\.

## Appendix BGuidelines

### B\.1Feedback Segmentation Annotation Guideline

The guideline used by annotators to segment instructor and LLM\-generated feedback into feedback units \(§[4\.1](https://arxiv.org/html/2607.00274#S4.SS1)\) is shown in Figure[10](https://arxiv.org/html/2607.00274#A2.F10); segmented examples appear in Table[2](https://arxiv.org/html/2607.00274#S4.T2)\.

![Refer to caption](https://arxiv.org/html/2607.00274v1/x6.png)Figure 10:Feedback segmentation annotation guideline \(page 1/1\)\.
### B\.2Feedback Unit Similarity Annotation Guideline

The guideline used to score similarity between feedback\-unit pairs on the 0–4 scale \(§[4\.2](https://arxiv.org/html/2607.00274#S4.SS2)\) is shown in Figures[11](https://arxiv.org/html/2607.00274#A2.F11)–[13](https://arxiv.org/html/2607.00274#A2.F13)\.

![Refer to caption](https://arxiv.org/html/2607.00274v1/x7.png)Figure 11:Feedback similarity annotation guideline \(page 1/3\)\.![Refer to caption](https://arxiv.org/html/2607.00274v1/x8.png)Figure 12:Feedback similarity annotation guideline \(page 2/3\)\.![Refer to caption](https://arxiv.org/html/2607.00274v1/x9.png)Figure 13:Feedback similarity annotation guideline \(page 3/3\)\.

## Appendix CFeedback Unit Similarity Pipeline

### C\.1Automatic similarity metrics\.

In our experiment and setting, lexical\-overlap measures such as BLEUPapineniet al\.\([2002](https://arxiv.org/html/2607.00274#bib.bib65)\)and chrFPopović \([2015](https://arxiv.org/html/2607.00274#bib.bib29)\)yield Pearson correlations below0\.10\.1with annotator scores\. Embedding\-based approaches \(MiniLMWanget al\.\([2020](https://arxiv.org/html/2607.00274#bib.bib28)\), MPNetSonget al\.\([2020](https://arxiv.org/html/2607.00274#bib.bib27)\), and BERTScoreZhanget al\.\([2019](https://arxiv.org/html/2607.00274#bib.bib60)\)\) perform better but still show low correlation with human scores, with Pearson correlations below0\.30\.3\. Cosine similarity using OpenAI’s embedding modeltext\-embedding\-3\-large161616[https://developers\.openai\.com/api/docs/models/text\-embedding\-3\-large](https://developers.openai.com/api/docs/models/text-embedding-3-large)is the strongest automatic baseline, but its performance depends heavily on the context provided to the embedding model\. Our best result \(r=0\.6696r=0\.6696,ρ=0\.6343\\rho=0\.6343\) is obtained by prefixing each feedback with “Semantic content of instructor feedback on a student\-written paragraph: ” before embedding\. Even so, it remains insufficiently aligned with human judgments for reliable evaluation\.

### C\.2Batch Similarity Scoring Details

#### Batched requests\.

Since the similarity guideline is long, scoring every pair independently for each paragraph, draft, and experimental setting would be expensive and inefficient\. We therefore score multiple pairs per request \(batch\_size=b=b\), instructing the model to evaluate each pair independently \(implementation details in §[C\.2](https://arxiv.org/html/2607.00274#A3.SS2)\)\. Batching reduces the cost in inverse proportion, but since it alters the prompting setting, we reverify the agreement with the human reference\. Correlation results for multiple models under the batching setup are reported in Table[3](https://arxiv.org/html/2607.00274#S4.T3)\(§[4\.2](https://arxiv.org/html/2607.00274#S4.SS2)\), where we adoptgemini\-3\.1\-flash\-lite\-previewfor the main pipeline on the strength of its agreement with human scores\.

#### Batching strategy\.

The similarity guideline used for scoring feedback\-unit pairs is nearly 2K tokens long \(although the exact token count depends on the tokenizer\), making repeated independent calls inefficient\. To reduce token cost, we evaluate multiple pairs in a single request \(batch\_size=b=b\), instructing the model to assign one score to each labeled pair independently\.

#### Token cost\.

LetAAdenote the irreducible per\-request token cost \(system instructions, formatting, and response overhead\), andBBthe guideline length\. When scoringbbpairs per request, the total token cost per pair can be approximated as

T​\(b\)=A\+Bb,T\(b\)=A\+\\frac\{B\}\{b\},so batching reduces the amortized guideline overhead by a factor ofO​\(1/b\)O\(1/b\)\.

#### Effect on agreement\.

Batching slightly changes the prompting context because the model evaluates multiple pairs within the same request\. We therefore verify that this does not materially affect similarity scoring\. Experiments withbatch\_size=50=50show agreement with the human reference comparable to the single\-pair setting reported in §[4\.2](https://arxiv.org/html/2607.00274#S4.SS2), indicating that batching does not meaningfully alter model judgments\.

#### Implementation notes\.

We avoided the Batch API \(i\.e\., offline requests used to reduce API cost\) in order to preserve the ability to retry malformed outputs\. In auxiliary runs withgpt\-5\-mini\-2025\-08\-07, which was not the primary model used in our main evaluation, we did not observe any syntactically invalid outputs\.gemini\-3\.1\-flash\-lite\-preview, which was used at much larger scale, produced a small number of syntactically invalid outputs, and in those cases, maximum of two retries were sufficient to obtain a syntactically valid output\.

## Appendix DThreshold\-Based Cross Matching

UniMatch’s default cross\-matching stage \(§[4\.3](https://arxiv.org/html/2607.00274#S4.SS3)\) aggregates similarity scores into soft precision and recall by treating each unit’s full self\-similarity as its maximum possible contribution\. We complement this with a threshold\-based variant that mirrors detection\-style evaluation in computer visionEveringhamet al\.\([2010](https://arxiv.org/html/2607.00274#bib.bib15)\): given a thresholdτ\\tauon the annotator\-aligned 0–4 similarity scale, a pair counts as a match only if its similarity is at leastτ\\tau, and the standard hard precision, recall, and F1follow\.

#### Formalization\.

LetG=\{Gi\}i=1\|G\|G=\\\{G\_\{i\}\\\}\_\{i=1\}^\{\|G\|\}andP=\{Pj\}j=1\|P\|P=\\\{P\_\{j\}\\\}\_\{j=1\}^\{\|P\|\}denote instructor and predicted feedback units for a single paragraph, with similarity scorerϕ​\(Gi,Pj\)∈\{0,1,2,3,4\}\\phi\(G\_\{i\},P\_\{j\}\)\\in\\\{0,1,2,3,4\\\}\. Forτ∈\{1,2,3,4\}\\tau\\in\\\{1,2,3,4\\\}, define the binarized adjacencyai​j​\(τ\)=𝟏​\[ϕ​\(Gi,Pj\)≥τ\]a\_\{ij\}\(\\tau\)=\\mathbf\{1\}\[\\phi\(G\_\{i\},P\_\{j\}\)\\geq\\tau\]and compute the maximum bipartite matching onaa:

Mτ∗=arg⁡maxM∈ℳ​∑\(Gi,Pj\)∈Mai​j​\(τ\),M^\{\*\}\_\{\\tau\}=\\arg\\max\_\{M\\in\\mathcal\{M\}\}\\sum\_\{\(G\_\{i\},P\_\{j\}\)\\in M\}a\_\{ij\}\(\\tau\),whereℳ\\mathcal\{M\}is the set of 1\-to\-1 matchings betweenGGandPP\. The match count and complementary error counts are

T​Pτ\\displaystyle TP\_\{\\tau\}=\|Mτ∗\|,\\displaystyle=\|M^\{\*\}\_\{\\tau\}\|,F​Pτ\\displaystyle FP\_\{\\tau\}=\|P\|−T​Pτ,\\displaystyle=\|P\|\-TP\_\{\\tau\},F​Nτ\\displaystyle FN\_\{\\tau\}=\|G\|−T​Pτ,\\displaystyle=\|G\|\-TP\_\{\\tau\},yielding the hard metrics

pτ=T​Pτ\|P\|,rτ=T​Pτ\|G\|,F1,τ=2​pτ​rτpτ\+rτ\.p\_\{\\tau\}=\\frac\{TP\_\{\\tau\}\}\{\|P\|\},\\quad r\_\{\\tau\}=\\frac\{TP\_\{\\tau\}\}\{\|G\|\},\\quad F\_\{1,\\tau\}=\\frac\{2\\,p\_\{\\tau\}\\,r\_\{\\tau\}\}\{p\_\{\\tau\}\+r\_\{\\tau\}\}\.

#### Interpretation ofτ\\tau\.

Because the similarity scale is grounded in the annotation guideline \(§[B\.2](https://arxiv.org/html/2607.00274#A2.SS2)\),τ\\tauhas a direct semantic reading:τ=1\\tau\{=\}1requires the pair to share at least a general feedback category,τ=2\\tau\{=\}2requires sharing an important point \(same targeted aspect or same comment\),τ=3\\tau\{=\}3requires that the units differ only in a minor detail, andτ=4\\tau\{=\}4requires near\-equivalence as feedback\. The threshold thus exposes how strict an “alignment” must be to count as feedback recovery\.

#### Sensitivity to over\-generation\.

In the soft variant, each additional predicted unit contributes its full self\-similarityϕmax\\phi\_\{\\max\}to the precision denominator∑jϕ​\(Pj,Pj\)\\sum\_\{j\}\\phi\(P\_\{j\},P\_\{j\}\), so a weak\-but\-nonzero match still inflates the denominator without comparably increasing the numerator\. The threshold\-based variant attenuates this: predictions affect precision only through their count, and any pair aboveτ\\taureceives full credit\. Weak\-but\-above\-threshold matches are therefore not partially discounted, and the per\-extra\-unit cost is smaller\. The variant offers a less stringent view when over\-generation is unavoidable; the soft variant remains preferred when fine\-grained similarity differences carry signal\.

#### Ranking robustness across the 74 configurations\.

Despite the qualitative difference in metric form, the relative ordering of the 74 configurations from §[5](https://arxiv.org/html/2607.00274#S5)is preserved under thresholding: for everyτ∈\{1,2,3,4\}\\tau\\in\\\{1,2,3,4\\\}, the Spearman rank correlation between configuration F1under the soft variant andF1,τF\_\{1,\\tau\}isρ=1\.0\\rho=1\.0\. The dominant findings reported in §[6](https://arxiv.org/html/2607.00274#S6)– the negative effect of verbosity, the gains from feedback\-category guidance, the limited contribution of rubric inclusion, and the secondary role of few\-shot prompting – therefore do not depend on the choice of matching metric\. We omit per\-configurationF1,τF\_\{1,\\tau\}tables, as the rank\-identity result subsumes them; the implementation is released alongside the soft variant for reproduction\.

#### Implementation\.

Threshold\-based matching reuses the same similarity matrices produced for the soft variant; no additional model calls are required\. We binarize the matrix atτ\\tauand run a bipartite matching solver on the resulting graph\. Note that the Hungarian algorithm handles the general case of weighted edges \(similarity scores\), whereas in this setting edges are unweighted\.

## Appendix EFull Evaluation Results

This appendix presents per\-configurationUniMatchscores for all 74 settings described in §[5](https://arxiv.org/html/2607.00274#S5)\. Tables[5](https://arxiv.org/html/2607.00274#A5.T5),[6](https://arxiv.org/html/2607.00274#A5.T6), and[7](https://arxiv.org/html/2607.00274#A5.T7)report precision, recall, F1, and number of generated feedback units under zero\-shot, few\-shot, and chain\-of\-thought \(CoT\) prompting, respectively\. Each table reports both the*single reference*evaluation \(paragraphs with exactly one instructor feedback unit\) and the*full corpus*evaluation \(every paragraph with at least one instructor annotation\); the two regimes track each other closely on every comparison reported in §[6](https://arxiv.org/html/2607.00274#S6), and we therefore discuss only the full\-corpus numbers throughout the main text\. Figure[16](https://arxiv.org/html/2607.00274#A5.F16)summarizes the F1effect of the prompt template across representative configurations\.

#### Per\-condition ablation visualizations\.

Figures[14](https://arxiv.org/html/2607.00274#A5.F14),[15\(a\)](https://arxiv.org/html/2607.00274#A5.F15.sf1), and[15\(b\)](https://arxiv.org/html/2607.00274#A5.F15.sf2)provide per\-model, per\-condition visualizations of the guidance, rubric, and few\-shot effects summarized in Table[4](https://arxiv.org/html/2607.00274#S6.T4)\(§[6](https://arxiv.org/html/2607.00274#S6)\)\.

#### Prompt template\.

The default constrained prompt \(V2V\_\{2\}, Figure[18](https://arxiv.org/html/2607.00274#A6.F18)\) substantially outperforms the unconstrained alternative \(V1V\_\{1\}, Figure[17](https://arxiv.org/html/2607.00274#A6.F17)\) across models, conditions, and shot settings \(Figure[16](https://arxiv.org/html/2607.00274#A5.F16)\)\. Across 28 matched pairs \(same model, guidance, rubric, and shot\),V2V\_\{2\}yields a mean F1gain of\+0\.090\+0\.090overV1V\_\{1\}, and wins in 27 of 28 pairs \(96%\)\. The mechanism is verbosity:V1V\_\{1\}averages roughly 12K generated feedback units per configuration vs\. 4\.6K forV2V\_\{2\}, and the resulting precision deficit overwhelms any recall benefit\. This is consistent with the broader verbosity finding in §[6](https://arxiv.org/html/2607.00274#S6): the constrained prompt enforces a per\-paragraph output budget that the unconstrained prompt does not\.

#### Few\-shot prompting underV1V\_\{1\}\.

Few\-shot exemplars contribute much more under the unconstrained prompt than under the constrained one\. In matched comparisons, few\-shot improves F1by\+0\.028\+0\.028underV1V\_\{1\}but only\+0\.005\+0\.005underV2V\_\{2\}\(Table[6](https://arxiv.org/html/2607.00274#A5.T6)\)\. This supports the interpretation in §[6](https://arxiv.org/html/2607.00274#S6)that exemplars are most useful when task framing is weakest:V2V\_\{2\}already specifies the target output structure, leaving little additional ground for exemplars to cover\. The few\-shot gain is precision\-driven across all 30 matched \(zero, few\) pairs: precision rises by\+0\.028\+0\.028on average while recall falls slightly \(−0\.004\-0\.004\)\.

#### Chain\-of\-thought\.

CoT prompting \(Table[7](https://arxiv.org/html/2607.00274#A5.T7)\) was evaluated on the three small models in the zero\-shot setting, using the unconstrainedV1V\_\{1\}prompt\. Under matched conditions, CoT improves F1by\+0\.041\+0\.041overV1V\_\{1\}but underperformsV2V\_\{2\}by−0\.056\-0\.056\. CoT settings produce 6\.6K–8\.4K generated feedback units per configuration, betweenV1V\_\{1\}andV2V\_\{2\}, and the qualitative patterns from the main results hold: Mistral is the strongest small model, feedback\-category guidance contributes a small positive effect, and rubric inclusion is essentially neutral\. CoT does not alter the overall story; like other settings, its position in the F1ordering is largely determined by how many feedback units it generates\.

0\.230\.230\.280\.280\.260\.260\.320\.320\.260\.260\.320\.320\.180\.180\.240\.240\.210\.210\.260\.260\.200\.200\.300\.30GBaseRFewDeepSeekMistralLlamaQwenLlamaGPTFigure 14:Effect of guidance on F1\. Each column is one model under a given condition \(Base: zero\-shot, no rubric;Few: few\-shot;R: rubric\); the top row adds feedback\-category guidance \(G\) and the bottom row omits it\. Guidance improves F1in every column\. Darker cells indicate higher F1\.0\.200\.200\.260\.260\.280\.280\.210\.210\.280\.280\.270\.27RBFewGLlamaMistralMistral\(a\)Rubric0\.260\.260\.320\.320\.370\.370\.270\.270\.310\.310\.360\.36FZGRLlamaQwenGPT\(b\)Few\-shot
Figure 15:Effect of rubric and few\-shot prompting on F1, each panel on its own color scale \(darker = higher F1\)\. Columns indicate a model and a generation setting; rows toggle the studied factor\.\(a\)Rubric present \(R, the assignment rubric\) vs\. absent \(B, base\), across few\-shot \(*Few*\), few\-shot\-with\-guidance, and guidance \(*G*, feedback categories in the prompt\)\.\(b\)Few\-shot \(F\) vs\. zero\-shot \(Z\), across guidance, guidance\-with\-rubric, and rubric; overlapping header lines mark the column with both settings\. Both factors shift F1only slightly and inconsistently, unlike the uniform gain from guidance \(Figure[14](https://arxiv.org/html/2607.00274#A5.F14)\)\.0\.180\.180\.230\.230\.250\.250\.320\.320\.320\.320\.120\.120\.130\.130\.170\.170\.140\.140\.240\.24V2V\_\{2\}V1V\_\{1\}BaseGRR\+GFew\+R\+GLlamaDeepSeekMistralGPTQwenFigure 16:F1across condition \(top header\) and prompt variant \(rows\)\.V1V\_\{1\}denotes the unconstrained prompt \(§[F](https://arxiv.org/html/2607.00274#A6)\);V2V\_\{2\}denotes the default single\-unit prompt used throughout the main experiments\.ShotGuidedRubricVModelSingle referenceFull corpusPRF1\#Seg\.PRF1\#Seg\.ZeroUn\-guidedNoRubric1Llama0\.0610\.2600\.099103090\.0800\.2320\.11915580DeepSeek0\.0660\.2160\.10177430\.0860\.1900\.11811571Mistral0\.0730\.2760\.11592730\.0970\.2480\.139137792Llama0\.1450\.2160\.17436290\.1830\.1830\.1835363DeepSeek0\.1360\.2270\.17040200\.1720\.1890\.1805851Mistral0\.2320\.3010\.26231610\.2510\.2210\.2354740Rubric1Llama0\.0580\.2670\.095112300\.0790\.2450\.11916763DeepSeek0\.0680\.2390\.10682250\.0950\.2200\.13211924Mistral0\.0880\.2980\.13682270\.1230\.2800\.17112212Qwen0\.0930\.2700\.13870750\.1120\.2390\.15311448GPT0\.1950\.3540\.25244200\.2260\.2850\.25367642Llama0\.1900\.2530\.21732510\.2160\.1920\.2034766Mistral0\.2250\.3490\.27337810\.2410\.2610\.2515798Qwen0\.3020\.3460\.32228000\.3120\.2390\.2714112GPT0\.3440\.3570\.35025260\.3640\.2500\.2963681GuidedNoRubric1Llama0\.0570\.2430\.093103510\.0810\.2340\.12015583DeepSeek0\.0640\.2500\.10290540\.0870\.2280\.12513208Mistral0\.0940\.3060\.14479380\.1270\.2880\.177121272Llama0\.2790\.3540\.31230940\.2930\.2470\.2684529DeepSeek0\.1960\.3160\.24236190\.2170\.2390\.2275326Mistral0\.2900\.3650\.32330690\.3010\.2590\.2784620Qwen0\.3220\.3460\.33426230\.3410\.2450\.2853859GPT0\.3270\.3460\.33625730\.3470\.2460\.2883792Rubric1Llama0\.0520\.2490\.087115930\.0740\.2390\.11317316DeepSeek0\.0610\.2400\.09789870\.0860\.2270\.12513467Mistral0\.0810\.2770\.12583590\.1140\.2630\.15912408Qwen0\.1150\.3030\.16764070\.1360\.2610\.17910305GPT0\.0720\.3150\.118106160\.0890\.2940\.136177392Llama0\.2560\.3340\.29031840\.2750\.2360\.2544592Mistral0\.2710\.3650\.31132860\.2880\.2670\.2774962Qwen0\.3730\.4030\.38726290\.3740\.2680\.3123838GPT0\.3560\.3870\.37126500\.3760\.2730\.3163892Table 5:UniMatchresults for all zero\-shot configurations on the single reference subset and the full corpus, as defined in §[5](https://arxiv.org/html/2607.00274#S5)\.V1V\_\{1\}denotes the unconstrained prompt \(Figure[17](https://arxiv.org/html/2607.00274#A6.F17)\);V2V\_\{2\}denotes the default single\-unit prompt \(Figure[18](https://arxiv.org/html/2607.00274#A6.F18)\), used throughout the main results\. P and R denote precision and recall as formulated in §[4\.3](https://arxiv.org/html/2607.00274#S4.SS3); \#Seg\. denotes the number of generated segments\. Model names are abbreviated for space; full names are given in §[5](https://arxiv.org/html/2607.00274#S5)\.ShotGuidedRubricVModelSingle referenceFull corpusPRF1\#Seg\.PRF1\#Seg\.FewUn\-GuidedNoRubric1Llama0\.0770\.2310\.11572780\.1020\.2030\.13610653DeepSeek0\.0700\.2370\.10879200\.0930\.2200\.13112097Mistral0\.1100\.2640\.15658190\.1480\.2410\.18387192Llama0\.1900\.2590\.21933270\.2180\.1980\.2084873DeepSeek0\.1940\.2760\.22834190\.2210\.2140\.2185113Mistral0\.2740\.3180\.29428180\.2930\.2270\.2564151Rubric1Llama0\.0660\.2290\.10284610\.0890\.2010\.12412062DeepSeek0\.0500\.2490\.083121420\.0680\.2290\.10517996Mistral0\.1000\.2820\.14768830\.1360\.2650\.18010440Qwen0\.2220\.3140\.26034430\.2570\.2490\.2535199GPT0\.2600\.3880\.31136410\.2990\.3100\.30555602Llama0\.2240\.2580\.24028080\.2350\.1800\.2044103Mistral0\.2410\.3440\.28334840\.2630\.2600\.2615305Qwen0\.2760\.3310\.30129190\.2920\.2340\.2604305GPT0\.3480\.3660\.35725570\.3690\.2560\.3023721GuidedNoRubric1Llama0\.0850\.2380\.12568470\.1170\.2210\.15310123DeepSeek0\.0500\.2630\.084128150\.0690\.2430\.10718947Mistral0\.0740\.3270\.121107200\.1030\.3030\.154157552Llama0\.3020\.3480\.32328120\.3040\.2310\.2624079DeepSeek0\.1980\.3040\.24034420\.2200\.2340\.2275293Mistral0\.2950\.3500\.32028910\.3090\.2490\.2754316Rubric1Llama0\.0660\.2120\.10177770\.0930\.1940\.12611183DeepSeek0\.0570\.2420\.092102860\.0810\.2280\.11915056Mistral0\.0980\.3050\.14975800\.1350\.2810\.18211198Qwen0\.2170\.2990\.25233380\.2510\.2370\.2445055GPT0\.2890\.3400\.31228670\.3180\.2590\.28543712Llama0\.2900\.3570\.32029930\.3020\.2450\.2714355Mistral0\.2530\.3410\.29032890\.2740\.2440\.2584774Qwen0\.3750\.4090\.39126570\.3770\.2740\.3173903GPT0\.3520\.3760\.36326040\.3690\.2630\.3073814Table 6:UniMatchresults for all few\-shot \(5\-shot\) configurations, evaluated on the same subsets and under the same prompt variants \(V1V\_\{1\},V2V\_\{2\}\) as the zero\-shot setting\. Column definitions follow Table[5](https://arxiv.org/html/2607.00274#A5.T5)\.ShotGuidedRubricVModelSingle referenceFull corpusPRF1\#Seg\.PRF1\#Seg\.ZeroUn\-GuidedNoRubricCoTLlama0\.1140\.2140\.14944640\.1490\.1870\.1666641DeepSeek0\.0980\.2210\.13647080\.1290\.2010\.1577322Mistral0\.1490\.2850\.19646530\.1820\.2400\.2077065RubricCoTLlama0\.1130\.2290\.15148570\.1450\.1950\.1677140DeepSeek0\.0930\.2060\.12843250\.1300\.1970\.1576894Mistral0\.1370\.3080\.18954840\.1730\.2680\.2108312GuidedNoRubricCoTLlama0\.1130\.2160\.14846150\.1430\.1840\.1616848DeepSeek0\.0960\.2000\.12942710\.1270\.1810\.1496629Mistral0\.1370\.2750\.18348760\.1820\.2450\.2097214RubricCoTLlama0\.1150\.2380\.15550010\.1460\.1980\.1687260DeepSeek0\.0910\.2040\.12645340\.1280\.1950\.1547111Mistral0\.1270\.2940\.17856310\.1650\.2590\.2028431Table 7:UniMatchresults for chain\-of\-thought configurations under the unconstrained prompt \(V1V\_\{1\}\)\. Column definitions follow Table[5](https://arxiv.org/html/2607.00274#A5.T5)\.

## Appendix FPrompt Templates

We use two prompt templates, both with two toggles \(Figures[17](https://arxiv.org/html/2607.00274#A6.F17),[18](https://arxiv.org/html/2607.00274#A6.F18)\):*Guided?*controls whether the feedback categories \(§[5](https://arxiv.org/html/2607.00274#S5)\) are listed, and*Rubric?*whether the assignment prompt and rubric are inserted via\{rubric\}\. The unconstrained template \(V1V\_\{1\}, Figure[17](https://arxiv.org/html/2607.00274#A6.F17)\) places no limit on the number of feedback units; the constrained template \(V2V\_\{2\}, Figure[18](https://arxiv.org/html/2607.00274#A6.F18)\) asks for a single most\-important point under a fixed output format\.V2V\_\{2\}is the default in the main experiments \(§[5](https://arxiv.org/html/2607.00274#S5)\);V1V\_\{1\}is used only in the prompt\-design comparison \(§[E](https://arxiv.org/html/2607.00274#A5)\)\.

![Refer to caption](https://arxiv.org/html/2607.00274v1/x10.png)Figure 17:Unconstrained prompt template \(free\-form paragraph\-level feedback with no constraint on the number of feedback units\)\. Used in the supplementary results comparing prompt designs \(§[E](https://arxiv.org/html/2607.00274#A5)\); the guided variant is introduced in §[5](https://arxiv.org/html/2607.00274#S5)\.![Refer to caption](https://arxiv.org/html/2607.00274v1/x11.png)Figure 18:Default single\-unit prompt template used throughout the main experiments \(§[5](https://arxiv.org/html/2607.00274#S5)\); the guided variant is also introduced §[5](https://arxiv.org/html/2607.00274#S5)\.