Llm Peer Review Eval

---
name: llm-peer-review-eval
description: Evaluates whether an LLM-based pairwise comparison framework can effectively identify high-impact academic papers compared to human peer review and traditional rating-based LLM methods. It probes the system's predictive accuracy for future scholarly influence, decision consistency with human committees, and susceptibility to biases in topic novelty and institutional representation. Use when the user wants to benchmark on OpenReview Conference Papers (ICLR, NeurIPS, CoRL, EMNLP), or asks about evaluating this task. Reports average_citation_count.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.11343
  bibtex_key: [REDACTED]
  confidence: high
---

# llm-peer-review-eval

> From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review — Zhang et al. (2025) (arXiv:2506.11343, 2025)

## What this evaluates

Evaluates whether an LLM-based pairwise comparison framework can effectively identify high-impact academic papers compared to human peer review and traditional rating-based LLM methods. It probes the system's predictive accuracy for future scholarly influence, decision consistency with human committees, and susceptibility to biases in topic novelty and institutional representation.

## Datasets

- **OpenReview Conference Papers (ICLR, NeurIPS, CoRL, EMNLP)** — total ?; splits: test (-1)

## Metrics

- `average_citation_count` **(primary)** — range: other
  - Mean number of citations received by papers accepted by the system. Used as the primary proxy for academic impact and system performance scaling.
- `decision_consistency` — range: percent
  - Percentage of papers where the GPT ranking system's decision (Oral, Spotlight, Poster, or Reject) matches the human peer review decision.
- `topic_novelty_distance` — range: other
  - Mean Euclidean distance between each paper's abstract embedding and its nearest neighbor within the same conference corpus. Smaller distances indicate lower novelty.
- `institutional_inequality_gini` — range: [0, 1]
  - Gini coefficient calculated over the distribution of first-author affiliations among top-tier accepted papers. Higher values indicate greater concentration at elite institutions.

## Input / output format

**Input**: Extracted paper content including title, abstract, figure/table captions, and main text, obtained via ScienceBeam from conference PDFs.

**Output**: Acceptance decision categorized into one of four tiers: Oral, Spotlight, Poster, or Reject, derived from aggregated pairwise comparison scores.

## Scoring recipe

```python
def compute_average_citation_count(predictions, gold_citations):
    accepted_citations = [gold_citations[pid] for pid in predictions if pid in gold_citations]
    if not accepted_citations:
        return 0.0
    return sum(accepted_citations) / len(accepted_citations)

def compute_decision_consistency(gpt_decisions, human_decisions):
    matches = sum(1 for g, h in zip(gpt_decisions, human_decisions) if g == h)
    return (matches / len(gpt_decisions)) * 100
```

## Common pitfalls

- Confusing the proposed pairwise ranking system with the GPT rating system baseline, which uses independent reviewers and a meta-reviewer instead of direct comparisons.
- Treating citation count as a direct quality metric without accounting for field-specific citation norms, conference tier differences, or citation lag across years.
- Assuming the embedding-based novelty distance perfectly captures conceptual innovation, as it only measures semantic similarity to existing abstracts.

## Evidence (verbatim from paper)

> We now empirically validate whether our proposed pairwise-based ranking framework better identifies high-impact papers compared to conventional rating-based approaches. Figure 2 shows the overall academic impact (measured by their average citation counts) of the accepted papers as the number of agents increases.

## Citation

```bibtex
@misc{zhang2025pairwisepeerreview,
  title={From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2506.11343}
}
```

- arXiv: 2506.11343