Peer Review Analysis Eval

---
name: peer-review-analysis-eval
description: This protocol evaluates the linguistic and content-level properties of academic peer review reports to assess how LLM assistance influences review quality, complexity, and aspect coverage over time. Use when the user wants to benchmark on ICLR & NeurIPS Peer Reviews, or asks about evaluating this task. Reports aspect_mentions.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.19578
  bibtex_key: wu2026impact
  confidence: high
---

# peer-review-analysis-eval

> Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI — Wu et al. (2026) (arXiv:2604.19578, 2026)

## What this evaluates

This protocol evaluates the linguistic and content-level properties of academic peer review reports to assess how LLM assistance influences review quality, complexity, and aspect coverage over time.

## Datasets

- **ICLR & NeurIPS Peer Reviews** — total ?; splits: ICLR 2017-2025 (-1), NeurIPS 2016-2024 (-1); repo https://github.com/njust-winchy/LLM_impact

## Metrics

- `text_length` — range: count
  - Average number of sentences and words per review report.
- `linguistic_complexity` — range: proportion
  - Measures lexical richness and syntactic indicators such as nominal subjects, auxiliary verbs, direct objects, and adverbial clauses per clause.
- `aspect_mentions` **(primary)** — range: mixed
  - Average length of aspect-related sentences, average number of aspect mentions (e.g., Clarity, Soundness, Originality), and sentiment polarity distribution for each aspect.
- `llm_detection_rate` — range: [0, 1]
  - Probability of a review being LLM-assisted estimated via maximum likelihood estimation.

## Input / output format

**Input**: Peer review reports from ICLR and NeurIPS conferences, including full text content and reviewer self-reported confidence scores (1-5).

**Output**: Computed statistics per year/conference: average text length, lexical/syntactic complexity metrics, aspect mention counts and sentiments, and LLM-assistance probability scores.

## Scoring recipe

```python
def compute_metrics(reviews):
    stats = {}
    for review in reviews:
        stats['text_len'] = count_sentences_and_words(review.text)
        stats['aspects'] = extract_aspect_mentions(review.text)
        stats['sentiment'] = compute_sentiment_polarity(review.text)
        stats['confidence'] = review.confidence_score
        stats['llm_prob'] = max_likelihood_estimate_llm_assistance(review.text)
    return aggregate_by_year_and_confidence(stats)
```

## Common pitfalls

- Temporal boundaries for 'pre-LLM' vs 'post-LLM' are defined by conference dates (ICLR 2023, NeurIPS 2022) rather than model release dates.
- Aspect mentions are identified via NLP pipelines rather than manual annotation, which may affect sentiment polarity accuracy.
- Confidence scores are self-reported by reviewers and may not correlate linearly with review quality or actual LLM usage.

## Evidence (verbatim from paper)

> Specifically, we analyze: The average length of aspect-related contents to measure elaboration depth (Figure 5). The average number of aspect mentions (e.g., Clarity, Soundness, Originality) to assess topic coverage (Figure 6). The average sentiment polarity associated with each aspect to explore tonal tendencies (Figure 7).

## Citation

```bibtex
@misc{wu2026impact,
  title={Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI},
  author={Wu et al. (2026)},
  year={2026},
  note={arXiv:2604.19578}
}
```

- arXiv: 2604.19578