R2pe Eval

---
name: r2pe-eval
description: Evaluates the ability to detect incorrect answers in chain-of-thought reasoning by analyzing inconsistencies across multiple reasoning paths. It measures how well intermediate steps can predict the correctness of a final answer without relying solely on the answer itself. Use when the user wants to benchmark on R2PE, or asks about evaluating this task. Reports Discernibility Score (DS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.10528
  bibtex_key: xu2024r2pe
  confidence: high
---

# r2pe-eval

> Can We Verify Step by Step for Incorrect Answer Detection? — Xu et al. (2024) (arXiv:2402.10528, 2024)

## What this evaluates

Evaluates the ability to detect incorrect answers in chain-of-thought reasoning by analyzing inconsistencies across multiple reasoning paths. It measures how well intermediate steps can predict the correctness of a final answer without relying solely on the answer itself.

## Datasets

- **R2PE** — total 38300; splits: full (38300); repo https://github.com/XinXU-USTC/R2PE

## Metrics

- `Discernibility Score (DS)` **(primary)** — range: [0, 1]
  - A numerical criterion that encapsulates the quality of rationales and answers across multiple reasoning paths. Predictions are made by comparing the DS against a threshold H (if DS < H, predict FALSE; else TRUE). Detection performance is subsequently evaluated using F1 score.

## Input / output format

**Input**: Question or claim Q, dataset name, queried LLM name, five CoT responses each containing a rationale r_i and extracted answer a_i, the final aggregated answer a, the ground-truth answer, and the ground-truth label L (TRUE/FALSE).

**Output**: Predicted label ̂L ∈ {TRUE, FALSE} based on whether the Discernibility Score (DS) falls below a threshold H.

## Scoring recipe

```python
predictions = []
for instance in dataset:
    ds = compute_discernibility_score(instance.responses)
    pred = 'FALSE' if ds < threshold_H else 'TRUE'
    predictions.append(pred)
tp = sum(1 for p, g in zip(predictions, dataset.labels) if p == 'TRUE' and g == 'TRUE')
fp = sum(1 for p, g in zip(predictions, dataset.labels) if p == 'TRUE' and g == 'FALSE')
fn = sum(1 for p, g in zip(predictions, dataset.labels) if p == 'FALSE' and g == 'TRUE')
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- The exact mathematical formula for the Discernibility Score (DS) is not provided in the benchmark construction section and must be retrieved from the paper's methodology.
- The benchmark aggregates answers via majority vote across 5 CoT generations per question before labeling, which can mask individual reasoning path errors.
- Evaluation is stratified into 45 subsets based on the combination of source dataset and generating LLM, so reporting aggregate scores without subset breakdown may hide significant performance variations.

## Evidence (verbatim from paper)

> We introduce a numerical criterion, the discernibility score (DS), to encapsulate the quality of the rationales and their corresponding answers. A low DS might suggest a potential mismatch between the final output a and the ground-truth answer. Hence, we will classify the example as false: ̂L=F, if its DS falls below a certain threshold H.

## Citation

```bibtex
@misc{xu2024r2pe,
  title={Can We Verify Step by Step for Incorrect Answer Detection?},
  author={Xu et al. (2024)},
  year={2024},
  note={arXiv:2402.10528}
}
```

- arXiv: 2402.10528