Mwp Value Accuracy Eval

---
name: mwp-value-accuracy-eval
description: Evaluates mathematical reasoning and robustness on single-equation math word problems. It probes a model's ability to parse linguistic variations, ignore irrelevant information, and solve inverted or structurally complex problems. Use when the user wants to benchmark on MAWPS, SVAMP, PARAMAWPS, or asks about evaluating this task. Reports Value accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.13899
  bibtex_key: raiyan2023mathwordproblem
  confidence: high
---

# mwp-value-accuracy-eval

> Math Word Problem Solving by Generating Linguistic Variants of Problem Statements — Raiyan et al. (2023) (arXiv:2306.13899, 2023)

## What this evaluates

Evaluates mathematical reasoning and robustness on single-equation math word problems. It probes a model's ability to parse linguistic variations, ignore irrelevant information, and solve inverted or structurally complex problems.

## Datasets

- **MAWPS** — total 2373; splits: 5-fold CV (2373)
- **SVAMP** — total 1000; splits: test (1000)
- **PARAMAWPS** — total 16278; splits: 5-fold CV (16278)

## Metrics

- `Value accuracy` **(primary)** — range: percent
  - Percentage of problems where the model's predicted numerical answer exactly matches the ground truth value. Calculated as (number of correct predictions / total number of problems) × 100.

## Input / output format

**Input**: Natural language math word problem statement.

**Output**: Numerical answer or mathematical expression. LLMs are instructed to output in the format 'Answer: [ANS]'.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_values):
    if str(pred).strip() == str(gold).strip():
        correct += 1
value_accuracy = (correct / len(predictions)) * 100
```

## Common pitfalls

- Uses 5-fold cross-validation for fine-tuned models but zero-shot evaluation for LLMs; results are not directly comparable across settings.
- PARAMAWPS contains adversarial variations (e.g., unrelated information, inverted questions) that test robustness beyond standard template matching.
- The reported accuracy often includes a majority voting mechanism over k generated variants; evaluating single instances without voting will yield lower scores.

## Evidence (verbatim from paper)

> Table 3: Value accuracy of the DeBERTa model and various baseline models. † denotes 5-fold cross validation. ... The DeBERTa model coupled with the Paraphrasing model and the Voting Mechanism outperforms all the baseline models in the MAwPS (Koncel-Kedziorski et al., 2016) dataset with an accuracy of 91.0%.

## Citation

```bibtex
@misc{raiyan2023mathwordproblem,
  title={Math Word Problem Solving by Generating Linguistic Variants of Problem Statements},
  author={Raiyan et al. (2023)},
  year={2023},
  note={arXiv:2306.13899}
}
```

- arXiv: 2306.13899