Spoken Mqa Eval

---
name: spoken-mqa-eval
description: Evaluates the ability of speech-based models to perform mathematical reasoning directly from audio inputs. It probes three distinct capabilities: direct arithmetic computation, contextual word-problem solving across single- and multi-step scenarios, and application of domain-specific mathematical knowledge in spoken language. Use when the user wants to benchmark on Spoken-MQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.15000
  bibtex_key: wei2025spokenmqa
  confidence: medium
---

# spoken-mqa-eval

> Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems — Wei et al. (2025) (arXiv:2505.15000, 2025)

## What this evaluates

Evaluates the ability of speech-based models to perform mathematical reasoning directly from audio inputs. It probes three distinct capabilities: direct arithmetic computation, contextual word-problem solving across single- and multi-step scenarios, and application of domain-specific mathematical knowledge in spoken language.

## Datasets

- **Spoken-MQA** — total 2769; splits: arithmetic (273), contextual_reasoning_single_step (594), contextual_reasoning_multi_step (1402), knowledge_oriented_reasoning (500)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of questions where the model's predicted answer exactly matches the ground truth answer string. Standard exact-match evaluation for mathematical QA benchmarks.

## Input / output format

**Input**: Audio recordings of mathematical problems spoken in natural language.

**Output**: Textual final answer (numerical value or short phrase).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if str(pred).strip().lower() == str(gold).strip().lower():
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Verbal ambiguity in spoken math expressions (e.g., 'nine factorial' vs 'nine exclamation mark') can cause misinterpretation without visual context.
- TTS synthesis artifacts or pronunciation errors may degrade performance independently of the model's actual reasoning capability.
- Cascade ASR-to-LLM pipelines often lose intermediate reasoning steps, leading to incorrect final answers even if the ASR transcription is accurate.

## Evidence (verbatim from paper)

> We introduce the Spoken-MQA benchmark, a curated collection of diverse textual math problems designed to evaluate spoken mathematical reasoning. As illustrated in Figure[1], the benchmark is structured into three categories based on the type of mathematical reasoning required: (1) Arithmetic, which emphasizes direct numerical computation; (2) Contextual Reasoning, involving single- or multi-step reasoning grounded in real-world scenarios; and (3) Knowledge-Oriented Reasoning, which requires the application of mathematical domain-specific knowledge.

## Citation

```bibtex
@misc{wei2025spokenmqa,
  title={Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems},
  author={Wei et al. (2025)},
  year={2025},
  note={arXiv:2505.15000}
}
```

- arXiv: 2505.15000