Iclr Ai Review Impact Eval

---
name: iclr-ai-review-impact-eval
description: Probes the causal impact of LLM-assisted peer reviews on paper scoring and acceptance outcomes at a major machine learning conference. It measures whether AI-assisted reviews systematically inflate scores and increase acceptance probabilities, particularly for borderline submissions. Use when the user wants to benchmark on ICLR Conference Reviews (2018-2024), or asks about evaluating this task. Reports acceptance_rate_difference.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.02150
  bibtex_key: russo2024aireviewlottery
  confidence: high
---

# iclr-ai-review-impact-eval

> The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates — Russo Latona et al. (2024) (arXiv:2405.02150, 2024)

## What this evaluates

Probes the causal impact of LLM-assisted peer reviews on paper scoring and acceptance outcomes at a major machine learning conference. It measures whether AI-assisted reviews systematically inflate scores and increase acceptance probabilities, particularly for borderline submissions.

## Datasets

- **ICLR Conference Reviews (2018-2024)** — total ?; splits: 2018-2023 (-1), 2024 (28028); repo https://github.com/epfl-dlab/AIRewiewLottery

## Metrics

- `mean_score_difference` — range: other
  - Average of (r_ai - r_h) across all valid review pairs per paper, where r_ai is AI-assisted score and r_h is human score.
- `acceptance_rate_difference` **(primary)** — range: percent
  - Coefficient β from linear regression y_k = α + β·L_k + γ·X_k on a matched sample, representing percentage point change in acceptance probability.
- `relative_probability_difference` — range: percent
  - From ordinal regression: probability that an AI-assisted review scores higher than a human review given they differ, minus 50%.

## Input / output format

**Input**: Per submission: review texts, review scores, abstract text, and final acceptance decision. Per review: text for LLM detection.

**Output**: Binary AI-assisted/human label per review; aggregated statistical estimates (mean score difference, regression coefficients for acceptance).

## Scoring recipe

```python
# 1. Classify reviews using GPTZero (prob < 0.5 => AI-assisted)
ai_reviews = [r for r in reviews if gptzero_prob(r) < 0.5]
human_reviews = [r for r in reviews if gptzero_prob(r) >= 0.5]

# 2. Compute mean score difference (Study 2)
diffs = []
for paper in papers_with_at_least_3_reviews:
    for ai_r in paper.ai_reviews:
        for h_r in paper.human_reviews:
            if h_r != reference_review:
                diffs.append(ai_r.score - h_r.score)
mean_diff = np.mean(diffs)

# 3. Compute acceptance_rate_difference (Study 3)
matched_pairs = match_submissions_by_review_count_and_scores()
beta = fit_linear_regression(matched_pairs, target='acceptance', features=['has_ai_review', 'controls']).coef['has_ai_review']
```

## Common pitfalls

- GPTZero's false-negative rate is uncorrected, making prevalence estimates a lower bound.
- Review scores are ordinal, yet treated as continuous for mean difference calculations.
- Matching relies on semantic similarity of abstracts/reviews, which may not fully control for paper quality or topic-specific reviewer biases.

## Evidence (verbatim from paper)

> We find that, on average, AI-assisted reviews were 0.14 points (95% CI [0.08, 0.19]) higher than human reviews. ... The fitted coefficients reveal that submissions that received AI-assisted reviews had 13.8% higher odds of being accepted compared to those that did not (p = 0.024), or alternatively, had 3.1 percentage points higher chances of being accepted (p = 0.024).

## Citation

```bibtex
@misc{russo2024aireviewlottery,
  title={The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates},
  author={Russo Latona et al. (2024)},
  year={2024},
  note={arXiv:2405.02150}
}
```

- arXiv: 2405.02150