Svamp Eval

---
name: svamp-eval
description: Evaluates whether NLP models can genuinely solve simple math word problems through arithmetic reasoning versus relying on shallow heuristics like bag-of-words matching or positional cues. It probes model brittleness by testing performance on standard datasets alongside carefully perturbed variants that remove questions or alter operator types. Use when the user wants to benchmark on MAWPS, ASDiv-A, SVAMP, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.07191
  bibtex_key: patel2021svamp
  confidence: high
---

# svamp-eval

> Are NLP Models really able to Solve Simple Math Word Problems? — Patel et al. (2021) (arXiv:2103.07191, 2021)

## What this evaluates

Evaluates whether NLP models can genuinely solve simple math word problems through arithmetic reasoning versus relying on shallow heuristics like bag-of-words matching or positional cues. It probes model brittleness by testing performance on standard datasets alongside carefully perturbed variants that remove questions or alter operator types.

## Datasets

- **MAWPS** — total ?; splits: (unstated)
- **ASDiv-A** — total ?; splits: (unstated)
- **SVAMP** — total ?; splits: (unstated); repo https://github.com/arkilpatel/SVAMP

## Metrics

- `accuracy` **(primary)** — range: percent
  - Exact-match correctness averaged over 5-fold cross-validation splits.

## Input / output format

**Input**: Natural language math word problem statements (sometimes modified to remove the question sentence or alter word order).

**Output**: A single numerical value representing the solution to the word problem.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if float(pred) == float(gold):
        correct += 1
accuracy = (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Models often achieve high accuracy on standard benchmarks by exploiting shallow heuristics (e.g., bag-of-words features or word order) rather than performing actual arithmetic.
- Performance drops significantly when the question sentence is removed or when problems are rephrased, indicating a lack of robust compositional reasoning.

## Evidence (verbatim from paper)

> The 5-fold cross-validation accuracies of the Transformer on MAWPS and ASDiv-A are provided in Table 16. The scores on Question-removed datasets are provided in Table 17 and on SVAMP challenge set is provided in Table 18.

## Citation

```bibtex
@misc{patel2021svamp,
  title={Are NLP Models really able to Solve Simple Math Word Problems?},
  author={Patel et al. (2021)},
  year={2021},
  note={arXiv:2103.07191}
}
```

- arXiv: 2103.07191