Review Quality Eval

---
name: review-quality-eval
description: Evaluates the quality of academic peer review reports across different conferences and years using a multi-dimensional framework. It measures how substantive, actionable, and well-grounded reviews are, and tracks whether these qualities decline over time. Use when the user wants to benchmark on Peer Review Campaigns (ICLR, NeurIPS, ACL), or asks about evaluating this task. Reports Q.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.15172
  bibtex_key: kuznetsov2026peerreview
  confidence: high
---

# review-quality-eval

> Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time — Kuznetsov et al. (2026) (arXiv:2601.15172, 2026)

## What this evaluates

Evaluates the quality of academic peer review reports across different conferences and years using a multi-dimensional framework. It measures how substantive, actionable, and well-grounded reviews are, and tracks whether these qualities decline over time.

## Datasets

- **Peer Review Campaigns (ICLR, NeurIPS, ACL)** — total ?; splits: full (-1); repo https://github.com/UKPLab/arxiv2026-review-quality-estimation

## Metrics

- `Q` **(primary)** — range: [0, 1]
  - Non-weighted average of all normalized measurement scores (LEN, ITX, EXL, GND, ACT, etc.) per review. Each base metric is first normalized to [0,1].
- `Spearman's ρ` — range: [-1, 1]
  - Rank-based correlation coefficient used to measure monotonic relationships between metric pairs and between lightweight vs LLM-based aggregate scores.

## Input / output format

**Input**: Raw peer review text (30 to 28k characters) associated with a submitted paper.

**Output**: Normalized scores for each quality dimension and an aggregate quality score Q per review.

## Scoring recipe

```python
# Normalize each base metric m to [0,1]
m_norm = (m - m_min) / (m_max - m_min)
# Compute aggregate score Q
Q = mean([m_norm for m in base_metrics])
# Correlation
rho = spearmanr(scores_A, scores_B)
# Significance test
p_val = bootstrap_test(Q_year1, Q_year2, n_iter=10000, ci=0.99, correction='bonferroni')
```

## Common pitfalls

- Count-based metrics (LEN, core items) have long tails and positive skew, requiring careful normalization before aggregation.
- Low Q scores may reflect non-genuine reviews (e.g., placeholder text, refusal to review, desk-reject flags) rather than poor quality.
- LLM-based metrics are computationally expensive and raise confidentiality concerns, so lightweight proxies are used for large-scale analysis.

## Evidence (verbatim from paper)

> Based on the effective ranges of the metrics, we normalize each metric to $[0,1]$ for ease of reporting and aggregation. Given that none of the metric pairs show extremely high correlations ($
ho$ > 0.9), we define an aggregate review quality score $Q$ as a non-weighted average of all our measurements (after normalization).

## Citation

```bibtex
@misc{kuznetsov2026peerreview,
  title={Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time},
  author={Kuznetsov et al. (2026)},
  year={2026},
  note={arXiv:2601.15172}
}
```

- arXiv: 2601.15172