Reviewer Too Eval

---
name: reviewer-too-eval
description: Evaluates the ability of LLM-based reviewer agents to predict conference acceptance decisions and generate high-quality peer reviews. It probes classification accuracy against human decisions and assesses review quality through LLM-judged pairwise comparisons across multiple dimensions. Use when the user wants to benchmark on ICLR-2k dataset, or asks about evaluating this task. Reports macro-F1 (5-way).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.08867
  bibtex_key: sahu2025reviewertoo
  confidence: high
---

# reviewer-too-eval

> ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review — Sahu et al. (2025) (arXiv:2510.08867, 2025)

## What this evaluates

Evaluates the ability of LLM-based reviewer agents to predict conference acceptance decisions and generate high-quality peer reviews. It probes classification accuracy against human decisions and assesses review quality through LLM-judged pairwise comparisons across multiple dimensions.

## Datasets

- **ICLR-2k dataset** — total 1963; splits: test (1963)

## Metrics

- `macro-F1 (5-way)` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score across five decision categories (Oral, Spotlight, Poster, Reject, Desk Reject), computed by averaging per-class F1 scores.
- `Accept/Reject Accuracy` — range: [0, 1]
  - Overall accuracy on the binary accept/reject classification task, calculated as the proportion of correctly predicted decisions.
- `ELO rating` — range: other
  - Pairwise win/loss/draw outcomes from LLM judges across five quality axes (Depth, Actionability, Summary, Clarity, Helpfulness), aggregated into an ELO score using standard logistic updates.

## Input / output format

**Input**: Manuscript text, optionally supplemented with conference guidelines, author rebuttals, and retrieved literature summaries depending on the baseline configuration.

**Output**: A predicted conference decision category (Oral, Spotlight, Poster, Reject, or Desk Reject) and/or a structured review text.

## Scoring recipe

```python
def compute_macro_f1(predictions, gold, classes):
    prec, rec, f1, _ = precision_recall_fscore_support(gold, predictions, average='macro', labels=classes)
    return f1

def compute_accuracy(predictions, gold):
    correct = sum(p == g for p, g in zip(predictions, gold))
    return correct / len(gold)
```

## Common pitfalls

- Withdrawn papers are merged into the Reject category, artificially inflating rejection rates compared to standard conference splits.
- Review quality relies on LLM judges rather than human experts, which may introduce bias in axes like Depth or Actionability.
- The dataset is stratified by original average reviewer scores, limiting generalizability to unreviewed or out-of-distribution submissions.

## Evidence (verbatim from paper)

> We assess alignment with real conference decisions by measuring both the 5-way classification performance (Oral, Spotlight, Poster, Reject, Desk Reject) and the binary Accept/Reject task; we report macro-averaged Precision, Recall, and F1, with macro averaging across classes $c$. We also report overall Accuracy, and False Positive Rate (for binary task).

## Citation

```bibtex
@misc{sahu2025reviewertoo,
  title={ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review},
  author={Sahu et al. (2025)},
  year={2025},
  note={arXiv:2510.08867}
}
```

- arXiv: 2510.08867