Reviewbench Eval

---
name: reviewbench-eval
description: Evaluates the quality and accuracy of automated peer reviews generated by LLMs. It probes both the semantic alignment of review text against paper-specific rubrics and the precision of predicted numerical ratings and acceptance decisions. Use when the user wants to benchmark on ReviewBench, or asks about evaluating this task. Reports Rubric Overall Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.14261
  bibtex_key: li2026reviewgrounder
  confidence: high
---

# reviewbench-eval

> ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents — Li et al. (2026) (arXiv:2604.14261, 2026)

## What this evaluates

Evaluates the quality and accuracy of automated peer reviews generated by LLMs. It probes both the semantic alignment of review text against paper-specific rubrics and the precision of predicted numerical ratings and acceptance decisions.

## Datasets

- **ReviewBench** — total ?; splits: test (-1)

## Metrics

- `Rubric Overall Score` **(primary)** — range: [-2, 14]
  - Sum of scores across eight paper-specific rubric dimensions (Core, Res, Comp, EBC, Clr, Cov, Tone, Contradict). Seven dimensions are scored on a 0–2 scale, while Contradict uses a -2 to 0 scale.
- `MSE` — range: [0, inf)
  - Mean Squared Error between predicted and ground-truth numerical ratings.
- `MAE` — range: [0, inf)
  - Mean Absolute Error between predicted and ground-truth numerical ratings.
- `ACC` — range: [0, 1]
  - Accuracy of predicted final decisions (e.g., accept/reject) against ground truth.
- `F1` — range: [0, 1]
  - F1 score for predicted final decisions against ground truth.

## Input / output format

**Input**: Paper text $p$. Models are strictly forbidden from accessing the aggregated reference review or paper-specific rubrics during generation.

**Output**: Generated review text, predicted numerical rating, and final decision.

## Scoring recipe

```python
def score_rubric(review):
    dims = ['Core', 'Res', 'Comp', 'EBC', 'Clr', 'Cov', 'Tone', 'Contradict']
    scores = [judge_dim(review, d) for d in dims]  # 0,1,2 scale except Contradict (-2,-1,0)
    return sum(scores)

def score_numeric(pred_rating, gold_rating, pred_dec, gold_dec):
    mse = mean((pred_rating - gold_rating)**2)
    mae = mean(abs(pred_rating - gold_rating))
    acc = mean(pred_dec == gold_dec)
    f1 = f1_score(gold_dec, pred_dec)
    return mse, mae, acc, f1
```

## Common pitfalls

- The 'Contradict' dimension uses a negative scoring scale (-2, -1, 0), unlike the other seven dimensions which use (0, 1, 2).
- Models are strictly forbidden from accessing the aggregated reference review or paper-specific rubrics during generation to prevent protocol leakage.
- Rubric-based evaluation emphasizes semantic quality over absolute scores, making it sensitive to adversarial instructions that can mislead models into ignoring guidelines.

## Evidence (verbatim from paper)

> We conduct evaluation on ReviewBench (Sec.[3]) using two complementary families of metrics: (1) Rubric-based Evaluation, which assesses the textual quality of generated reviews across eight paper-specific rubric dimensions (Sec.[3.2]); and (2) Numeric-field Evaluation, which measures predicted ratings with MSE/MAE and decisions with ACC/F1 (Sec.[3.3]).

## Citation

```bibtex
@misc{li2026reviewgrounder,
  title={ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.14261}
}
```

- arXiv: 2604.14261