Extra Cot Eval

---
name: extra-cot-eval
description: Evaluates the ability of large language models to generate mathematically reasoned chain-of-thought outputs that are compressed to a target token budget while preserving logical fidelity and answer accuracy. Use when the user wants to benchmark on GSM8K, MATH-500, AMC2023, or asks about evaluating this task. Reports Acc@all.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.08324
  bibtex_key: tang2026extracot
  confidence: high
---

# extra-cot-eval

> Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression — Yuntian Tang et al. (arXiv:2602.08324, 2026)

## What this evaluates

Evaluates the ability of large language models to generate mathematically reasoned chain-of-thought outputs that are compressed to a target token budget while preserving logical fidelity and answer accuracy.

## Datasets

- **GSM8K** — total ?; splits: test (-1)
- **MATH-500** — total ?; splits: test (-1)
- **AMC2023** — total ?; splits: test (-1)

## Metrics

- `Acc@all` **(primary)** — range: percent
  - Accuracy computed over the entire test set by comparing the model's extracted final answer to the ground truth label.
- `ActRatio` — range: [0, 1]
  - The realized compression ratio aggregated over the dataset, calculated as the mean of compressed-to-original token ratios strictly within <think>...</think> blocks, averaged over parsable outputs.

## Input / output format

**Input**: A mathematical reasoning problem or question.

**Output**: A compressed chain-of-thought reasoning trace enclosed in <think>...</think> tags, followed by the final answer.

## Scoring recipe

```python
def score(predictions, golds, original_tokens):
    correct = 0
    act_ratios = []
    for pred, gold, orig in zip(predictions, golds, original_tokens):
        if pred.strip() == gold.strip():
            correct += 1
        compressed_tokens = count_tokens_in_think_block(pred)
        original_think_tokens = count_tokens_in_think_block(orig)
        if original_think_tokens > 0:
            act_ratios.append(compressed_tokens / original_think_tokens)
    accuracy = correct / len(golds)
    act_ratio = sum(act_ratios) / len(act_ratios) if act_ratios else 0
    return accuracy, act_ratio
```

## Common pitfalls

- Token counts and ActRatio are computed exclusively within <think>...</think> blocks, excluding the final answer tokens.
- ActRatio is averaged only over outputs that are successfully parsed for <think> blocks; unparsable outputs are excluded from the ratio calculation.
- Models may exhibit 'control collapse' where they ignore the target compression ratio to prioritize accuracy, leading to ActRatio significantly exceeding the target.

## Evidence (verbatim from paper)

> Evaluation protocol. We evaluate all methods across five target compression ratios: γ∈{0.2,0.4,0.6,0.8,1.0}, and the <COMP_POLICY> mode. Our evaluation centers on two primary metrics: accuracy and compression efficiency. Accuracy (Acc@all) is computed over the entire test set. To measure compression efficiency, we report the Actual Ratio (ActRatio), which is the realized compression ratio aggregated over the dataset. Crucially, all token counts and ratios are computed using think-only accounting, measuring tokens strictly within <think>...</think> blocks. Consequently, ActRatio and its underlying token counts are averaged over parsable outputs.

## Citation

```bibtex
@misc{tang2026extracot,
  title={Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression},
  author={Yuntian Tang et al.},
  year={2026},
  note={arXiv:2602.08324}
}
```

- arXiv: 2602.08324