Longcot Eval

---
name: longcot-eval
description: Probes a model's ability to maintain coherent state, plan, and execute multi-step reasoning over long, interdependent chains of thought spanning tens to hundreds of thousands of tokens across domains like chemistry, mathematics, and chess. Use when the user wants to benchmark on LongCoT, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.14140
  bibtex_key: motwani2026longcot
  confidence: high
---

# longcot-eval

> LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning — Motwani et al. (2026) (arXiv:2604.14140, 2026)

## What this evaluates

Probes a model's ability to maintain coherent state, plan, and execute multi-step reasoning over long, interdependent chains of thought spanning tens to hundreds of thousands of tokens across domains like chemistry, mathematics, and chess.

## Datasets

- **LongCoT** — total 2500; splits: test (2500)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of questions where the final answer matches the expected answer after sequential verification (RegEx, flexible RegEx, LLM extraction, or manual check).

## Input / output format

**Input**: Single-shot problem descriptions requiring navigation of complex dependency graphs (DAGs), with reasoning budgets up to provider limits (e.g., 128K tokens).

**Output**: Chain-of-thought reasoning trace followed by a final answer in the expected format.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, g in zip(predictions, gold):
        # Sequential verification pipeline
        answer = extract_answer(pred) # RegEx -> flexible RegEx -> LLM fallback
        if answer == g: # Manual verification fallback if needed
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Assuming pass@k or self-consistency can be easily scaled; high token costs prevent these experiments.
- Expecting domain-specific knowledge to drive performance; success relies on long-horizon state maintenance and planning, not deep domain expertise.
- Assuming accuracy degradation is solely due to context window saturation; the benchmark shows performance drops sharply as DAG complexity increases, independent of context limits.

## Evidence (verbatim from paper)

> Final answers are verified through sequential checks: RegEx on expected format, flexible RegEx on full responses if needed, and LLM-based extraction (GPT-5-mini) as a fallback. These answers are then manually verified for correctness. We find that performance is uniformly low, with GPT 5.2 achieving the highest accuracy of 9.83% followed by Gemini 3 Pro (6.08%) and Grok 4.1 Fast Reasoning (2.04%).

## Citation

```bibtex
@misc{motwani2026longcot,
  title={LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning},
  author={Motwani et al. (2026)},
  year={2026},
  note={arXiv:2604.14140}
}
```

- arXiv: 2604.14140