State Tracking Eval

---
name: state-tracking-eval
description: Evaluates whether LLMs can track dynamic states over sequential update instructions. It probes the model's ability to maintain and update internal representations of an environment's state across multiple steps, testing sequential reasoning and input-window memory limits. Use when the user wants to benchmark on State-Tracking-Tasks (LinearWorld, HandSwap, Lights), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.10457
  bibtex_key: rezaee2025exploring
  confidence: high
---

# state-tracking-eval

> Exploring State Tracking Capabilities of Large Language Models — Rezaee et al. (2025) (arXiv:2511.10457, 2025)

## What this evaluates

Evaluates whether LLMs can track dynamic states over sequential update instructions. It probes the model's ability to maintain and update internal representations of an environment's state across multiple steps, testing sequential reasoning and input-window memory limits.

## Datasets

- **State-Tracking-Tasks (LinearWorld, HandSwap, Lights)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly answered queries across all depth levels and tasks. Calculated as the number of correct predictions divided by the total number of test instances.

## Input / output format

**Input**: Natural language prompt containing an initial state description, a sequence of update instructions (state changes), and a final query about the current state.

**Output**: Direct answer to the query (e.g., a value, boolean, or short phrase), optionally preceded by a Chain-of-Thought reasoning trace if CoT prompting is used.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_label in zip(predictions, gold):
        if str(pred).strip().lower() == str(gold_label).strip().lower():
            correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Models may default to answering based solely on the initial state rather than tracking updates, which the authors explicitly control for with a 'Stateless' baseline.
- Performance degradation at depth ≥ 2 for smaller models is gradual, not binary, so averaging across all depths can mask early-stage failures.
- Chain-of-Thought (CoT) prompting alters the input format by appending reasoning traces, which increases context length and may conflate state-tracking ability with input-window memory limits.

## Evidence (verbatim from paper)

> Table 1: The main evaluation results of systems on different tasks reported in terms of accuracy at various depths.

## Citation

```bibtex
@misc{rezaee2025exploring,
  title={Exploring State Tracking Capabilities of Large Language Models},
  author={Rezaee et al. (2025)},
  year={2025},
  note={arXiv:2511.10457}
}
```

- arXiv: 2511.10457