Peerprism Eval

---
name: peerprism-eval
description: Evaluates the ability of various LLM text detection methods to distinguish between human-written and AI-generated peer reviews, while also assessing their robustness to hybrid (human idea + AI text) workflows. Use when the user wants to benchmark on PeerPrism, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.14513
  bibtex_key: sadeghian2026peerprism
  confidence: high
---

# peerprism-eval

> PeerPrism: Peer Evaluation Expertise vs Review-writing AI — Sadeghian et al. (2026) (arXiv:2604.14513, 2026)

## What this evaluates

Evaluates the ability of various LLM text detection methods to distinguish between human-written and AI-generated peer reviews, while also assessing their robustness to hybrid (human idea + AI text) workflows.

## Datasets

- **PeerPrism** — total 20690; splits: test (-1); repo https://github.com/Reviewerly-Inc/PeerPrism

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard binary classification accuracy: the proportion of correctly classified reviews (Human vs. AI) out of the total evaluated instances.

## Input / output format

**Input**: Peer review texts labeled as original human-written, fully synthetic LLM-generated, or hybrid/transformed.

**Output**: Binary classification label (Human or AI) or probability score, depending on the detector.

## Scoring recipe

```python
# For each detector:
scores = detector.predict(reviews)
if scores are probabilities:
    threshold = calibrate_threshold(scores, held_out_subset)
    preds = [1 if s >= threshold else 0 for s in scores]
else:
    preds = scores
accuracy = sum(pred == gold for pred, gold in zip(preds, gold_labels)) / len(gold_labels)
```

## Common pitfalls

- Hybrid reviews (human ideas + AI text) are explicitly excluded from threshold calibration and standard accuracy computation, but used separately for robustness checks.
- Detectors are evaluated in their original off-the-shelf configurations without fine-tuning on the PeerPrism dataset.
- Score-based methods require threshold calibration on a balanced held-out subset before evaluation, as they lack predefined thresholds.

## Evidence (verbatim from paper)

> For binary evaluation, we define two strict ground-truth classes: original human-written reviews (Human) and fully synthetic LLM-generated reviews (AI). Hybrid regimes are excluded from threshold calibration and standard accuracy computation and are instead used to assess robustness under mixed-provenance conditions. For detectors that output probabilities, we follow the thresholds recommended in their original implementations. For score-based methods without prescribed thresholds (Anchor and Lastde++), we calibrate the decision boundary on a balanced held-out subset and fix it for all subsequent experiments. We report accuracy and confusion matrices separately for each provenance regime.

## Citation

```bibtex
@misc{sadeghian2026peerprism,
  title={PeerPrism: Peer Evaluation Expertise vs Review-writing AI},
  author={Sadeghian et al. (2026)},
  year={2026},
  note={arXiv:2604.14513}
}
```

- arXiv: 2604.14513