Implicit Cot Eval

---
name: implicit-cot-eval
description: Evaluates a model's ability to perform arithmetic and grade-school math reasoning without generating explicit intermediate chain-of-thought steps. It measures both the exact-match accuracy of the final answer and the inference speed relative to a no-CoT baseline. Use when the user wants to benchmark on Multi-digit multiplication, GSM8K, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.14838
  bibtex_key: [REDACTED]
  confidence: high
---

# implicit-cot-eval

> From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step — Deng et al. (2024) (arXiv:2405.14838, 2024)

## What this evaluates

Evaluates a model's ability to perform arithmetic and grade-school math reasoning without generating explicit intermediate chain-of-thought steps. It measures both the exact-match accuracy of the final answer and the inference speed relative to a no-CoT baseline.

## Datasets

- **Multi-digit multiplication** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1); HF `gsm8k`

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Exact match accuracy of producing the final answer. Calculated as the number of correctly predicted final answers divided by the total number of test instances.
- `Speed` — range: other
  - Inference throughput measured in examples per second on an Nvidia H100 GPU with batch size 1, normalized by the speed of the corresponding No CoT baseline model.

## Input / output format

**Input**: Arithmetic problems (e.g., K-by-K multiplication equations or grade-school math word problems).

**Output**: A single final numerical answer or solution string, without intermediate reasoning steps.

## Scoring recipe

```python
def compute_metrics(predictions, golds, model, baseline_model):
    # Accuracy
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
    accuracy = correct / len(golds)
    # Speed (normalized)
    speed_raw = count_examples_per_second(model, batch_size=1, device='H100')
    speed_baseline = count_examples_per_second(baseline_model, batch_size=1, device='H100')
    speed_normalized = speed_raw / speed_baseline
    return {'accuracy': accuracy, 'speed_normalized': speed_normalized}
```

## Common pitfalls

- Implicit CoT models do not output reasoning traces, so step-level evaluation metrics are inapplicable.
- Speed comparisons must be normalized to the No-CoT baseline on identical hardware and batch size; raw examples/second are not comparable across setups.
- GSM8K results use augmented training data from Deng et al., which may differ from standard splits or prompts.

## Evidence (verbatim from paper)

> Accuracy (Acc in the table) measures the exact match accuracy of producing the final answer. Speed measures the number of examples per second during inference using a batch size of 1, normalized by the speed of the corresponding No CoT model.

## Citation

```bibtex
@misc{deng2024implicitcot,
  title={From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step},
  author={Deng et al. (2024)},
  year={2024},
  note={arXiv:2405.14838}
}
```

- arXiv: 2405.14838