Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review

# Integrating Author Expertise and Intent in Responses to Peer Review
Source: https://arxiv.org/html/2602.11173

## Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review

Qian Ruan, Iryna Gurevych
Ubiquitous Knowledge Processing Lab (UKP Lab)
Department of Computer Science and Hessian Center for AI (hessian.AI)
Technical University of Darmstadt
www.ukp.tu-darmstadt.de

###### Abstract

Author response (rebuttal) writing is a critical stage of scientific peer review that demands substantial author effort. In practice, authors possess domain expertise, author-only information, and response strategies – concrete forms of author expertise and intent – and seek NLP assistance that integrates these signals into author response generation (ARG). Yet this author-in-the-loop paradigm lacks formal NLP formulation and systematic study: no dataset provides fine-grained author signals, existing ARG work lacks author inputs and controls, and no evaluation measures response reflection of author signals and effectiveness in addressing reviewer concerns. To fill these gaps, we introduce (i) Re3Align, the first large-scale dataset of aligned review–response–revision triplets, where revisions proxy author signals; (ii) REspGen, an author-in-the-loop ARG framework supporting flexible author input, multi-attribute control, and evaluation-guided refinement; and (iii) REspEval, a comprehensive evaluation suite with 20+ metrics spanning input utilization, controllability, response quality, and discourse. Experiments with SOTA LLMs demonstrate the benefits of author input and evaluation-guided refinement, the impact of input specificity on response quality, and controllability–quality trade-offs. We release our dataset,¹ generation and evaluation tools.²

¹https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/4982
²https://github.com/UKPLab/acl2026-respgen-respeval

## 1 Introduction

Author response (rebuttal) writing is a critical stage of scientific peer review, where authors address reviewer concerns to seek favorable acceptance decisions. This process demands substantial author effort, making it a promising yet challenging NLP task (Kuznetsov et al. 2024). In practice, authors write rebuttals drawing on domain expertise and strategic intent. Many reviewer concerns can only be addressed using information known exclusively to the authors, such as planned clarifications (e.g., the precise definition of script evaluation in Figure 1), newly conducted experiments, and design rationales. Beyond content, authors strategically choose response approaches (revising, justifying, deferring, or promising future work) and control attributes such as length, tone, and discourse structure. Yet prior NLP research treats author response generation (ARG) as a generic, review-only text generation problem, failing to formalize this practice. We bridge this gap by reformulating ARG as an **author-in-the-loop** task.

To realize this, two fundamental questions remain unanswered. (1) **How should author-in-the-loop ARG be formulated and benchmarked, and how should author expertise and intent be provided to generation systems?** Existing ARG studies rely solely on reviewer comments, lacking author expertise and intent (Purkayastha et al. 2023; Tan et al. 2024; Zhang et al. 2025), and a systematic study supported by data and generation frameworks is missing. (2) **Does the generated response reflect the author's provided expertise and intent while effectively addressing reviewer concerns?** This requires evaluating response quality, factual grounding, input coverage, and adherence to constraints and plans. Prior ARG work relies on surface similarity or coarse quality criteria, leaving these dimensions unsupported. Controllable ARG remains unexplored, and rigorous evaluation of controllability and trade-offs is an open challenge even in controllable text generation broadly (Zhang et al. 2023; Liang et al. 2024).

Author-in-the-loop ARG study is further hindered by data scarcity. Collecting author signals during live rebuttals is rarely feasible due to practical and ethical constraints. One alternative is approximating author signals using paper revisions: in conference settings, responses describe planned changes that later appear in revised papers, enabling post-hoc edit extraction to proxy author signals at response time; in journal settings, responses argue based on already-implemented revisions. This requires **complete paper records** encompassing original and revised manuscripts, peer reviews, and authentic author responses, which are available in only a few existing datasets (Dyck et al. 2023; Lin et al. 2023). More critically, existing resources lack the fine-grained annotations needed to model author signals at the granularity of individual reviewer concerns, such as edit analyses, review–response segment alignments, and mappings to concrete paper edits.

To address these challenges, we introduce **Re3Align** (§3), the first large-scale dataset for author-in-the-loop ARG, comprising 3.4k complete paper records with 440k sentence-level edit annotations and 15k aligned review–response–edit triplets. We further propose **REspGen** (§4), an author-in-the-loop ARG framework supporting various levels of author input specificity and enabling controllable generation over response planning and length, with iterative refinement guided by **REspEval** (§5). REspEval provides over 20 novel metrics spanning controllability, input utilization (factual grounding and coverage), response quality (targeting, specificity, and convincingness), and discourse characteristics (tone–stance profiles and transitions). Finally, we experiment with five SOTA LLMs across nine settings to systematically analyze the effects of author signals, input specificity, attribute control, and evaluation-guided refinement (§6).

Our work makes four key contributions:

- The first large dataset of review–response–edit triplets with rich annotations, enabling a new formulation of the ARG task;
- An author-in-the-loop ARG framework supporting flexible author input, multi-attribute control, and evaluation-guided refinement;
- A comprehensive evaluation suite with 20+ novel metrics for controllability, input utilization, response quality, and discourse;
- Extensive experiments across five LLMs and nine settings, yielding insights into ARG behavior under varied inputs and controls, and cross-dimensional trade-offs.

This work provides the first systematic formalization and study of author-in-the-loop response generation and evaluation, bridging author expertise and intent with NLP assistance to support effective and efficient author response writing.

| | Data | Generation | Evaluation |
|---|---|---|---|
| | review | response | dimension | metric |
| Prior work | ✓ seg | ✓ seg | ✗ | ✗ | ✗ | ✗ | Similarity |
| | ✓ doc | ✓ doc | ✗ | ✗ | ✗ | ✗ | Similarity |
| | ✓ doc | ✓ doc | ✗ | ✗ | ✗ | ✗ | |
| Ours | ✓ seg | ✓ seg | ✓ sent | Similarity, Quality, Discourse, Input Utilization, Controllability |

**Table 1:** Comparison of related works on author response generation, including data, generation task formulations and evaluation dimensions and metrics. doc/seg/sent: document-/segment-/sentence-level alignments and annotations.

## 2 Related Work

**Author Response Generation** has recently emerged as a challenging and underexplored task in NLP for scientific peer review (Kuznetsov et al. 2024; Staudinger et al. 2024). Early work on author response includes argument-pair extraction (Cheng et al. 2020) and response discourse analysis (Kennard et al. 2022). Empirical studies further identify key success factors for effective responses, including explicit revision statements, high specificity, concrete evidence, and appropriate tone (Noble 2017; Gao et al. 2019; Huang et al. 2023). Recent work shifts toward generation, with studies on attitude- and theme-guided generation (Purkayastha et al. 2023) and multi-turn review–rebuttal dialogue (Tan et al. 2024; Zhang et al. 2025). However, these generation approaches rely solely on reviewer comments, producing generic responses that lack concrete details, especially those requiring author expertise. Evaluation is limited to similarity metrics, overlooking response diversity and broader success factors from empirical studies.

As summarized in Table 1, we address these limitations in three ways by (i) introducing the first large-scale triplet dataset of reviews, responses, and aligned sentence-level edits, treating revisions as explicit signals of author expertise and intent; (ii) formulating ARG as an author-in-the-loop task integrating author expertise and intent through explicit input and controllable generation; (iii) proposing a comprehensive evaluation suite with 20+ metrics spanning four dimensions beyond similarity-based evaluation.

**Controllable Text Generation and Evaluation** aims to steer model outputs toward user-specified constraints (Zhang et al. 2023). Prior work primarily focuses on single-attribute control, including length (Kikuchi et al. 2016), topic (Wang et al. 2019), and sentiment (Firdaus et al. 2020), as well as content-based control such as query-focused (Xu and Lapata 2021), entity-centric (Maddela et al. 2022), and aspect-based generation (Li et al. 2023). Recent surveys highlight persistent challenges in simultaneous multi-attribute control, trade-offs between controllability and generation quality, and the lack of rigorous evaluation methods (Zhang et al. 2023; Liang et al. 2024). In ARG, controllability remains unexplored despite authors' need to strategically control response construction while integrating their own content. We provide the first study of controllability in ARG, examining control over length, discourse planning, and content integration. We further introduce a comprehensive evaluation framework with fine-grained metrics assessing: (i) how well generations adhere to single- and simultaneous multi-attribute controls; (ii) how effectively author-provided content is incorporated; and (iii) how response quality is impacted.

## 3 Dataset Construction: Re3Align

### 3.1 Data Collection and Preprocessing

Our framing of ARG requires raw data capturing the full review–revision–response (Re3) process with authentic human texts. Only a few resources, such as the EMNLP24 subset of NLPEERv2 (Dyck et al. 2023) and MOPRD (Lin et al. 2023), provide peer reviews, author responses, original submissions, and revised papers. EMNLP24 provides peer reviews and rebuttal discussions from OpenReview,³ which we organize into reviewer–author discussion chains, extracting and merging consecutive author replies into single responses. MOPRD offers data from PeerJ⁴ across multiple scientific domains including computer science, chemistry, physics, and materials science. We retain only papers with a complete Re3 record. The final corpus includes 679 EMNLP24 papers and 2,715 PeerJ papers (Table 2), covering both conference and journal workflows. We group each paper's versions, reviews, and responses under a unified identifier and convert them into intertextual graphs (ITGs) (Kuznetsov et al. 2022), augmented with sentence-level nodes (details in §A.1).

| | #Paper | #Pair | #Edit | #Linked Edit | #Re3Triplet |
|---|---|---|---|---|---|
| EMNLP24 | 679 | 2,108 | 86,247 | 76,621 | 1,933 |
| PeerJ | 2,715 | 13,963 | 353,551 | 181,534 | 13,588 |
| **Total** | **3,394** | **16,071** | **439,798** | **198,296** | **15,521** |

**Table 2:** Re3Align Dataset Statistics. Reported are the counts of papers, aligned review–response pairs, annotated sentence-level edits, edits linked to the pairs, and the final number of aligned triplets.

### 3.2 Review-Response Pair Alignment and Revision Annotation

Authors often quote review sentences to structure their replies. To extract review–response pairs, we match every review sentence to every response sentence using an assembled matching algorithm (§A.2) and merge the longest contiguous matches to identify quoted review spans. These spans are then used to segment the response, with each segment defined as the text following a quoted span and preceding the next one. An illustrative example is shown in Figure 4 in §A.2. After applying quality filtering strategies (§A.2), we obtain 2,108 and 13,963 review–response segment pairs from EMNLP24 and PeerJ, respectively. Human verification of 100 pairs confirms a 98% alignment accuracy. We further apply SOTA revision analysis models (Ruan et al. 2024a,b) to align sentence-level edits across paper versions and label each with edit action and intent. These models achieve over 90 F1 for alignment and action labeling, and 84.3 F1/85.6% accuracy for edit intent classification. In total, this produces 439,798 edits.

### 3.3 Re3 Triplet Alignment

For each submission, we have the original paper D^t, the revised paper D^(t+1), and reviewer–author exchanges (C_k, A_k) for reviewer k. Sentences in D^t and D^(t+1) are denoted x_j^t and x_i^(t+1). From earlier steps, we extract sentence-level edits e_ij = e(x_i^(t+1), x_j^t), with the f