Echochain Eval

---
name: echochain-eval
description: Evaluates how voice assistants handle mid-generation interruptions by testing their ability to revise in-progress responses while maintaining context and switching objectives. It probes state-update reasoning, contextual inertia, interruption amnesia, and objective displacement under full-duplex interaction conditions. Use when the user wants to benchmark on EchoChain, or asks about evaluating this task. Reports pass_fail.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.16456
  bibtex_key: modi2026echochain
  confidence: high
---

# echochain-eval

> EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions — Modi et al. (2026) (arXiv:2604.16456, 2026)

## What this evaluates

Evaluates how voice assistants handle mid-generation interruptions by testing their ability to revise in-progress responses while maintaining context and switching objectives. It probes state-update reasoning, contextual inertia, interruption amnesia, and objective displacement under full-duplex interaction conditions.

## Datasets

- **EchoChain** — total ?; splits: test (-1)

## Metrics

- `pass_fail` **(primary)** — range: [0, 1]
  - Binary evaluation based on instance-specific rubrics. Calculated as the number of model responses that meet the rubric criteria divided by the total number of evaluated instances.

## Input / output format

**Input**: Full conversation transcripts and aligned synthetic audio, including initial user prompts, mid-generation interruptions, and the model's post-interruption response.

**Output**: Binary pass/fail label per model response, with failed responses additionally categorized into a predefined failure taxonomy (contextual inertia, interruption amnesia, objective displacement).

## Scoring recipe

```python
def compute_pass_fail(predictions, rubrics):
    passes = 0
    for pred, rubric in zip(predictions, rubrics):
        if pred.satisfies(rubric):
            passes += 1
    return passes / len(predictions)
```

## Common pitfalls

- Rubrics are instance-specific and manually authored, introducing potential annotator subjectivity or inconsistency across different conversation contexts.
- Audio quality artifacts from synthetic speech may unfairly penalize models if reviewers conflate generation errors with state-update reasoning failures.
- Blinding is only applied during the rubric application stage; initial screening is done by a single rater per conversation, risking bias before model identity is hidden.

## Evidence (verbatim from paper)

> Using this rubric, annotators assign a binary pass/fail label for every model response; failed responses that reflect interruption-driven errors are then labeled using the benchmark failure taxonomy.

## Citation

```bibtex
@misc{modi2026echochain,
  title={EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions},
  author={Modi et al. (2026)},
  year={2026},
  note={arXiv:2604.16456}
}
```

- arXiv: 2604.16456