Gpt4 Based Exact Match

---
name: gpt4-based-exact-match
description: Evaluates whether a language model's final numerical answer to a grade-school math word problem matches the ground truth. It uses an external LLM to extract the final answer from the model's generated solution and compares it against the gold answer. Use when the user has predictions and gold and needs to compute GPT4-based-Exact-Match.
metadata:
  skill_kind: metric
  source_arxiv: 2402.14830
  bibtex_key: mitra2024orcamath
  confidence: high
---

# gpt4-based-exact-match

> Orca-Math: Unlocking the potential of SLMs in Grade School Math — Mitra et al. (2024) (arXiv:2402.14830, 2024)

## What this evaluates

Evaluates whether a language model's final numerical answer to a grade-school math word problem matches the ground truth. It uses an external LLM to extract the final answer from the model's generated solution and compares it against the gold answer.

## Datasets

- **GSM8K** — total ?; splits: test (-1)

## Metrics

- `GPT4-based-Exact-Match` **(primary)** — range: [0, 1]
  - An LLM (GPT-4) is prompted to extract the final numerical answer from the model's generated solution and compare it to the gold answer. If they match, the prediction is marked Correct; otherwise, Incorrect.

## Input / output format

**Input**: Model-generated solution text, problem question, and gold solution text.

**Output**: A structured response containing 'Error Analysis' (one sentence comparing answers) and 'Final Verdict' (Correct or Incorrect).

## Scoring recipe

```python
def compute_metric(predictions, gold_answers):
    correct_count = 0
    for pred, gold in zip(predictions, gold_answers):
        # GPT-4 extracts final answer from pred and compares to gold
        verdict = call_gpt4_extract_and_compare(pred, gold)
        if verdict == "Correct":
            correct_count += 1
    return correct_count / len(predictions)
```

## Common pitfalls

- Relies on GPT-4 for answer extraction, which may introduce latency/cost and potential extraction errors compared to regex-based exact match.
- Only checks the final numerical answer, ignoring intermediate reasoning steps or alternative valid solution paths.

## Evidence (verbatim from paper)

> We use exact match as the metric. Given a model generated answer, we prompt GPT4 to extract the final short answer and match it with the gold short answer. We will refer to this metric as, GPT4-based-Exact-Match.

## Citation

```bibtex
@misc{mitra2024orcamath,
  title={Orca-Math: Unlocking the potential of SLMs in Grade School Math},
  author={Mitra et al. (2024)},
  year={2024},
  note={arXiv:2402.14830}
}
```

- arXiv: 2402.14830