Squeez Tool Output Pruning Eval

---
name: squeez-tool-output-pruning-eval
description: Evaluates a model's ability to extract the smallest verbatim evidence block from a single tool observation given a focused query, while maximizing compression and correctly returning empty output for true negatives. Use when the user wants to benchmark on Squeez Benchmark, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.04979
  bibtex_key: [REDACTED]
  confidence: high
---

# squeez-tool-output-pruning-eval

> Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents — Kovács (2026) (arXiv:2604.04979, 2026)

## What this evaluates

Evaluates a model's ability to extract the smallest verbatim evidence block from a single tool observation given a focused query, while maximizing compression and correctly returning empty output for true negatives.

## Datasets

- **Squeez Benchmark** — total 11477; splits: test (618); repo https://github.com/KRLabsOrg/squeez

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Token-level F1 score computed between the predicted verbatim span and the gold span. Calculated as 2 * (precision * recall) / (precision + recall).
- `Recall` — range: [0, 1]
  - Token-level recall: fraction of gold tokens correctly predicted.
- `Precision` — range: [0, 1]
  - Token-level precision: fraction of predicted tokens that appear in the gold span.
- `Exact` — range: [0, 1]
  - Exact match accuracy: 1 if the predicted string exactly equals the gold string, else 0.
- `Strict F1` — range: [0, 1]
  - Exact span match F1: 1 if prediction exactly matches gold, else 0. Often reported alongside token-level F1 to penalize partial matches.
- `Compression` — range: [0, 1]
  - Fraction of input tokens removed: 1 - (|prediction| / |gold|). Higher values indicate more aggressive pruning.

## Input / output format

**Input**: A focused query string and a raw tool observation (e.g., git log, service log, docker logs, build output, kubectl status) provided as a single prompt.

**Output**: A verbatim subset of lines from the tool observation, or an empty string if no relevant content exists.

## Scoring recipe

```python
def compute_metrics(pred, gold):
    pred_tokens = pred.split()
    gold_tokens = gold.split()
    exact = 1.0 if pred == gold else 0.0
    common = Counter(pred_tokens) & Counter(gold_tokens)
    num_same = sum(common.values())
    prec = num_same / len(pred_tokens) if pred_tokens else 0.0
    rec = num_same / len(gold_tokens) if gold_tokens else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    compression = 1.0 - (len(pred_tokens) / len(gold_tokens)) if gold_tokens else 0.0
    return {'Exact': exact, 'Precision': prec, 'Recall': rec, 'F1': f1, 'Compression': compression}
```

## Common pitfalls

- Heuristic baselines like BM25 fail because relevance depends on the query rather than lexical overlap alone.
- Models frequently select semantically adjacent but incorrect blocks in repetitive logs or Git history.
- Zero-shot models often generate explanatory text for true negatives instead of returning empty output.

## Evidence (verbatim from paper)

> Squeez-2B attains the highest recall among all systems while maintaining 92% compression. It outperforms the 18× larger Qwen 3.5 35B A3B by 11 recall points and the unfine-tuned 2B base by 33 points.

## Citation

```bibtex
@misc{kovacs2026squeez,
  title={Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents},
  author={Kovács (2026)},
  year={2026},
  note={arXiv:2604.04979}
}
```

- arXiv: 2604.04979