Mistake Finding Eval

---
name: mistake-finding-eval
description: Evaluates an LLM's ability to detect and locate the first logical error in a multi-step chain-of-thought reasoning trace. It probes whether models can accurately identify specific reasoning steps that contain mistakes or correctly assert that a trace is entirely correct. Use when the user wants to benchmark on BIG-Bench Mistake, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.08516
  bibtex_key: tyen2023mistakefinding
  confidence: high
---

# mistake-finding-eval

> LLMs cannot find reasoning errors, but can correct them given the error location — Tyen et al. (2023) (arXiv:2311.08516, 2023)

## What this evaluates

Evaluates an LLM's ability to detect and locate the first logical error in a multi-step chain-of-thought reasoning trace. It probes whether models can accurately identify specific reasoning steps that contain mistakes or correctly assert that a trace is entirely correct.

## Datasets

- **BIG-Bench Mistake** — total 2186; splits: test (2186); repo https://github.com/WHGTyen/BIG-Bench-Mistake

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact match accuracy: 1 if the predicted step number matches the gold step number, or if both predict 'no mistake'; 0 otherwise. Averaged across all instances.
- `weighted F1 score` — range: [0, 1]
  - Weighted average F1 score for binary correctness prediction (mistake vs no mistake), weighted by the prevalence of each label in the dataset.

## Input / output format

**Input**: A chain-of-thought reasoning trace. Depending on the prompting method, this is either the full trace (trace-level), or the partial trace up to a specific step (step-level).

**Output**: A single integer N representing the step number of the first mistake, or the string 'No'/'No mistake' if the trace is correct. For step-level prompting, a binary Yes/No per step.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- Step-level prompting requires multiple generation calls per trace, which significantly increases error rates compared to trace-level prompting due to accumulated generation probability.
- Binary correctness F1 scores can be misleadingly high due to class imbalance; predicting all traces as incorrect yields a baseline weighted F1 of 78.
- Traces are generated by PaLM 2 L, so evaluating them with GPT models may introduce cross-model bias not present in self-evaluation settings.

## Evidence (verbatim from paper)

> Table 4 shows the accuracy of GPT-4-Turbo, GPT-4, and GPT-3.5-Turbo on our mistake-finding dataset. For each question, the possible answers are either that there are no mistakes, or, if there is a mistake, the number N indicating the step in which the first mistake occurs. A model's output is only considered correct if the location matches exactly, or the output correctly indicates that there are no mistakes.

## Citation

```bibtex
@misc{tyen2023mistakefinding,
  title={LLMs cannot find reasoning errors, but can correct them given the error location},
  author={Tyen et al. (2023)},
  year={2023},
  note={arXiv:2311.08516}
}
```

- arXiv: 2311.08516