Thinkswitcher Math Eval

---
name: thinkswitcher-math-eval
description: Evaluates a model's ability to dynamically switch between short and long chain-of-thought reasoning modes based on task complexity, balancing mathematical problem-solving accuracy against computational efficiency. It probes whether a single reasoning model can adaptively select concise or elaborate reasoning paths without architectural changes or post-training. Use when the user wants to benchmark on GSM8K, MATH-500, AIME24, AIME25, LiveAoPS, Omni-MATH-500, OlympiadBench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.14183
  bibtex_key: liang2025thinkswitcher
  confidence: high
---

# thinkswitcher-math-eval

> ThinkSwitcher: When to Think Hard, When to Think Fast — Liang et al. (2025) (arXiv:2505.14183, 2025)

## What this evaluates

Evaluates a model's ability to dynamically switch between short and long chain-of-thought reasoning modes based on task complexity, balancing mathematical problem-solving accuracy against computational efficiency. It probes whether a single reasoning model can adaptively select concise or elaborate reasoning paths without architectural changes or post-training.

## Datasets

- **GSM8K** — total ?; splits: test (-1)
- **MATH-500** — total ?; splits: test (-1)
- **AIME24** — total ?; splits: test (-1)
- **AIME25** — total ?; splits: test (-1)
- **LiveAoPS** — total ?; splits: test (-1)
- **Omni-MATH-500** — total ?; splits: test (-1)
- **OlympiadBench** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly solved problems out of the total number of test instances. Calculated as correct predictions divided by total questions.
- `Token Count` — range: tokens
  - Average number of generated tokens per question across the dataset. Reflects computational cost and varies based on the selected short or long CoT mode.
- `AUC-AC` — range: [0, 1]
  - Area under the curve plotting model accuracy against token efficiency across varying decision thresholds. Quantifies the trade-off performance of the adaptive mechanism.
- `nAUC-AC` — range: [0, 1]
  - Normalized variant of AUC-AC that measures the advantage gained by the adaptive mechanism over a linear interpolation between short-CoT-only and long-CoT-only performance points.

## Input / output format

**Input**: Natural language math word problems or competition-level math questions.

**Output**: Chain-of-thought reasoning (either short or long form) followed by a final answer. The model internally selects the reasoning mode based on a learned switcher.

## Scoring recipe

```python
def compute_metrics(predictions, gold_answers, token_counts):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if extract_final_answer(p) == g)
    accuracy = correct / len(gold_answers)
    avg_tokens = sum(token_counts) / len(token_counts)
    return {'accuracy': accuracy, 'avg_tokens': avg_tokens}
```

## Common pitfalls

- The 'Avg.' column in Table 1 reports macro-averaged accuracy and token counts across all seven benchmarks, not a dataset-weighted average.
- Training data (MATH train, GSM8K train, historical AIME, Omni-MATH subset) is explicitly excluded from test sets to prevent data leakage; evaluators must strictly use the designated test splits.
- Token count is not a fixed hyperparameter but a per-instance metric that varies with the selected reasoning mode; averaging must be done over all test instances, not per-mode.

## Evidence (verbatim from paper)

> To evaluate the overall trade-off performance, we use the AUC-AC to quantify how well ThinkSwitcher maintains accuracy across its spectrum of token efficiencies, and its normalized variant nAUC-AC to measure the advantage gained by ThinkSwitcher’s adaptive mechanism over a linear interpolation between SC-Only and LC-Only performance points.

## Citation

```bibtex
@misc{liang2025thinkswitcher,
  title={ThinkSwitcher: When to Think Hard, When to Think Fast},
  author={Liang et al. (2025)},
  year={2025},
  note={arXiv:2505.14183}
}
```

- arXiv: 2505.14183