Theoremqa Eval

---
name: theoremqa-eval
description: Evaluates LLMs' ability to apply domain-specific theorems from mathematics, physics, computer science, and finance to solve complex scientific problems. It probes theorem-driven reasoning, numerical computation, and program generation capabilities. Use when the user wants to benchmark on TheoremQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.12524
  bibtex_key: chen2023theoremqa
  confidence: high
---

# theoremqa-eval

> TheoremQA: A Theorem-driven Question Answering dataset — Chen et al. (2023) (arXiv:2305.12524, 2023)

## What this evaluates

Evaluates LLMs' ability to apply domain-specific theorems from mathematics, physics, computer science, and finance to solve complex scientific problems. It probes theorem-driven reasoning, numerical computation, and program generation capabilities.

## Datasets

- **TheoremQA** — total 800; splits: test (800)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. The model's final output is compared against the ground truth answer, with exact matching or numerical tolerance applied depending on the answer type.

## Input / output format

**Input**: A question requiring theorem-driven reasoning across STEM domains. For multimodal variants, an image is provided and converted to a text caption.

**Output**: A final answer in the required format (integer, float, boolean, list, or multiple-choice option), or a Python program to compute the answer.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Models often generate correct logical steps in Chain-of-Thought or Program-of-Thoughts but make minor calculation errors in intermediate steps, leading to incorrect final answers.
- Multimodal questions rely on BLIP-generated captions, which cause significant information loss for diagrammatic inputs, making them nearly impossible for text-only models.
- Simply concatenating theorem definitions as additional prompt context yields negligible accuracy gains (<1%) due to the abstract, symbolic nature of theorems.

## Evidence (verbatim from paper)

> With CoT prompting, GPT-3 (text-davinci-002) and GPT-3.5 models are only achieving 16.6% and 22.8% accuracy. By adopting the program as the intermediate reasoning form, both models can gain reasonable improvements.

## Citation

```bibtex
@misc{chen2023theoremqa,
  title={TheoremQA: A Theorem-driven Question Answering dataset},
  author={Chen et al. (2023)},
  year={2023},
  note={arXiv:2305.12524}
}
```

- arXiv: 2305.12524