Reviewmt Eval

---
name: reviewmt-eval
description: Evaluates LLMs on simulating the academic peer review process across multi-turn dialogues. It probes the model's ability to generate relevant paper summaries, write comprehensive reviews, and make accurate acceptance or rejection decisions based on long-context interactions. Use when the user wants to benchmark on ReviewMT, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.05688
  bibtex_key: tan2024reviewmt
  confidence: high
---

# reviewmt-eval

> Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions — Tan et al. (2024) (arXiv:2406.05688, 2024)

## What this evaluates

Evaluates LLMs on simulating the academic peer review process across multi-turn dialogues. It probes the model's ability to generate relevant paper summaries, write comprehensive reviews, and make accurate acceptance or rejection decisions based on long-context interactions.

## Datasets

- **ReviewMT** — total ?; splits: train (-1), test (100); repo https://github.com/chengtan9907/ReviewMT

## Metrics

- `Paper hit rate` — range: percent
  - Percentage of test instances where the generated paper summary exactly matches or sufficiently overlaps with the ground truth summary.
- `Review hit rate` — range: percent
  - Percentage of test instances where the generated review text exactly matches or sufficiently overlaps with the ground truth review.
- `Decision hit rate` — range: percent
  - Percentage of test instances where the model's predicted acceptance/rejection decision exactly matches the ground truth decision.
- `MAE` — range: other
  - Mean Absolute Error between the model's predicted numerical score and the ground truth score. Only reported for the multi-turn ICLR subset.
- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for the binary decision classification task (accept vs. reject).

## Input / output format

**Input**: Paper text and multi-turn dialogue history (author responses, reviewer comments, decision maker feedback).

**Output**: Generated paper summary, review text, and acceptance/rejection decision.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    n = len(preds)
    paper_hits = sum(1 for p, g in zip(preds['paper'], golds['paper']) if exact_match(p, g))
    review_hits = sum(1 for p, g in zip(preds['review'], golds['review']) if exact_match(p, g))
    decision_hits = sum(1 for p, g in zip(preds['decision'], golds['decision']) if p == g)
    maes = [abs(p - g) for p, g in zip(preds['score'], golds['score'])]
    f1 = f1_score(golds['decision'], preds['decision'], average='macro')
    return {
        'paper_hit_rate': paper_hits / n,
        'review_hit_rate': review_hits / n,
        'decision_hit_rate': decision_hits / n,
        'mae': sum(maes) / n,
        'f1_score': f1
    }
```

## Common pitfalls

- Zero-shot models show very low review and decision hit rates, requiring supervised fine-tuning for competitive performance.
- MAE and review hit rates are only evaluated on the multi-turn ICLR subset, not the one-turn NC subset.
- Some models (e.g., Yuan) have strict safety constraints that prevent them from outputting decisions, artificially lowering their decision hit rate.

## Evidence (verbatim from paper)

> As shown in Table[2], most LLMs demonstrate high paper hit rates, indicating their ability to generate relevant content related to the papers. However, zero-shot performance reveals lower review hit rates and decision hit rates, suggesting that LLMs struggle to provide scores and decisions. Supervised fine-tuning significantly improves performance across all metrics.

## Citation

```bibtex
@misc{tan2024reviewmt,
  title={Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions},
  author={Tan et al. (2024)},
  year={2024},
  note={arXiv:2406.05688}
}
```

- arXiv: 2406.05688