Iclr2025 Review Feedback Eval

---
name: iclr2025-review-feedback-eval
description: Evaluates the causal impact of LLM-generated feedback on peer review quality by measuring reviewer engagement, revision rates, feedback incorporation, and downstream rebuttal dynamics in a large-scale randomized controlled trial. Use when the user wants to benchmark on ICLR 2025 Reviews, or asks about evaluating this task. Reports update_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.09737
  bibtex_key: thakkar2025canllmfeedbackenhance
  confidence: high
---

# iclr2025-review-feedback-eval

> Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 — Nitya Thakkar et al. (2025) (arXiv:2504.09737, 2025)

## What this evaluates

Evaluates the causal impact of LLM-generated feedback on peer review quality by measuring reviewer engagement, revision rates, feedback incorporation, and downstream rebuttal dynamics in a large-scale randomized controlled trial.

## Datasets

- **ICLR 2025 Reviews** — total 44831; splits: control (22364), feedback (22467); repo https://github.com/zou-group/review_feedback_agent

## Metrics

- `update_rate` **(primary)** — range: percent
  - Percentage of reviews that are edited after receiving feedback. An update is defined as an edit distance greater than 5 between the initial and modified review. For the control group, updates must occur at least one hour after the initial post to account for feedback latency.
- `incorporation_rate` — range: percent
  - Proportion of LLM-generated feedback items that are clearly integrated into the reviewer's modified review, measured via an LLM-based pipeline validated against human annotations.
- `review_length_change` — range: words
  - Average difference in word count between initial and modified reviews, calculated across the summary, strengths, weaknesses, and questions sections.
- `acceptance_rate` — range: percent
  - Percentage of papers accepted in the control group versus the feedback group, where groups are defined by whether any review for the paper was selected to receive feedback.

## Input / output format

**Input**: Initial peer review text (summary, strengths, weaknesses, questions) and paper metadata.

**Output**: Modified peer review text, updated numerical scores, and subsequent rebuttal/reply text.

## Scoring recipe

```python
def calculate_update_rate(initial_reviews, modified_reviews):
    updated_count = 0
    for init, mod in zip(initial_reviews, modified_reviews):
        if edit_distance(init, mod) > 5:
            updated_count += 1
    return updated_count / len(initial_reviews) * 100

def calculate_incorporation_rate(feedback_items, modified_reviews):
    incorporated = llm_pipeline.check_integration(feedback_items, modified_reviews)
    return incorporated / len(feedback_items) * 100
```

## Common pitfalls

- Using a strict edit distance threshold (<5) to filter out minor typo fixes or score-only changes, which significantly affects update rate calculations.
- Failing to account for the intent-to-treat design, where 7.9% of the feedback group did not actually receive feedback, diluting the measured effect size.
- Comparing control group updates without enforcing a >1 hour delay, which would conflate natural late edits with feedback-induced updates.

## Evidence (verbatim from paper)

> With an update rate difference of roughly 17 percentage points (Figure 3B), we can see that reviews that received feedback were much more likely to be updated than those that did not.

## Citation

```bibtex
@misc{thakkar2025canllmfeedbackenhance,
  title={Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025},
  author={Nitya Thakkar et al. (2025)},
  year={2025},
  note={arXiv:2504.09737}
}
```

- arXiv: 2504.09737