Slump Eval

---
name: slump-eval
description: Measures how much final code fidelity degrades when a system's design is progressively disclosed through multi-turn interaction rather than provided upfront. It evaluates semantic faithfulness to a committed design and structural integration of dependencies in long-horizon coding agents. Use when the user wants to benchmark on SLUMP benchmark, or asks about evaluating this task. Reports IF50.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.17104
  bibtex_key: yan2026slump
  confidence: high
---

# slump-eval

> When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents — Lu Yan et al. (2026) (arXiv:2603.17104, 2026)

## What this evaluates

Measures how much final code fidelity degrades when a system's design is progressively disclosed through multi-turn interaction rather than provided upfront. It evaluates semantic faithfulness to a committed design and structural integration of dependencies in long-horizon coding agents.

## Datasets

- **SLUMP benchmark** — total 20; splits: test (20)

## Metrics

- `MCF` — range: [0, 4]
  - Mean Component Faithfulness. Computes the average rubric score $s_i \in \{0,1,2,3,4\}$ across all $N$ components in a paper: $\frac{1}{N} \sum_{i=1}^N s_i$.
- `DIR` — range: [0, 1]
  - Dependency Integration Ratio. Measures structural integration by dividing the number of public symbols actually reused by downstream code by the total number of exported symbols: $|E_{used}| / |E_{total}|$.
- `IF50` **(primary)** — range: [0, 1]
  - Implementation Faithfulness. Normalized summary metric combining semantic faithfulness and structural integration: $\frac{1}{2}(\frac{MCF}{4}) + \frac{1}{2}DIR$. Serves as the primary endpoint metric for SLUMP.

## Input / output format

**Input**: Multi-turn interaction script (emergent condition) or complete paper text in a single prompt (single-shot control).

**Output**: Final integrated code repository implementing the target design.

## Scoring recipe

```python
def compute_metrics(code, components):
    # MCF: rubric score 0-4 per component
    mcf_scores = [rubric_score(comp) for comp in components]
    MCF = sum(mcf_scores) / len(components)
    
    # DIR: count exported vs reused public symbols
    E_total = count_public_symbols_exported(code)
    E_used = count_public_symbols_reused_in_downstream(code)
    DIR = E_used / E_total if E_total > 0 else 0.0
    
    # IF50: primary endpoint
    IF50 = 0.5 * (MCF / 4) + 0.5 * DIR
    return MCF, DIR, IF50
```

## Common pitfalls

- Executable test pass rates do not reliably detect faithfulness loss, as they may validate the produced code without verifying alignment with the intended design.
- Faithfulness loss occurs during final integration of previously implemented components, not merely during isolated implementation.

## Evidence (verbatim from paper)

> We report three metrics. Mean Component Faithfulness (MCF) measures semantic faithfulness to the committed design. Let $s_i \in \{0,1,2,3,4\}$ be the rubric score for component $i$ and let $N$ be the number of components in the paper. Then MCF = 1/N \sum_{i=1}^N s_i. MCF ranges from 0 to 4. Dependency Integration Ratio (DIR) (Le Hai et al., 2025) measures structural integration: DIR = |E_used| / |E_total|, where $E_{total}$ is the set of public symbols exported by standalone modules and $E_{used}$ is the subset actually reused by downstream integration code. DIR ranges from 0 to 1. To summarize both dimensions, we define implementation faithfulness IF50 = 1/2 (MCF/4) + 1/2 DIR. IF50 ranges from 0 to 1 and serves as our primary endpoint metric for SLUMP.

## Citation

```bibtex
@misc{yan2026slump,
  title={When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents},
  author={Lu Yan et al. (2026)},
  year={2026},
  note={arXiv:2603.17104}
}
```

- arXiv: 2603.17104