Shredbench Eval

---
name: shredbench-eval
description: Evaluates multimodal LLMs' ability to reconstruct shredded documents from fragmented visual inputs. It probes cross-modal semantic reasoning, visual discontinuity alignment, and fine-grained positional continuity awareness across natural language, source code, and tabular data. Use when the user wants to benchmark on ShredBench, or asks about evaluating this task. Reports NED.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.23813
  bibtex_key: guo2026shredbench
  confidence: high
---

# shredbench-eval

> ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction — Guo et al. (2026) (arXiv:2604.23813, 2026)

## What this evaluates

Evaluates multimodal LLMs' ability to reconstruct shredded documents from fragmented visual inputs. It probes cross-modal semantic reasoning, visual discontinuity alignment, and fine-grained positional continuity awareness across natural language, source code, and tabular data.

## Datasets

- **ShredBench** — total ?; splits: test (-1); repo https://github.com/ythere-y/ShredBench

## Metrics

- `NED` **(primary)** — range: [0, 1]
  - Normalized Edit Distance. Computed as the Levenshtein distance between prediction Ŷ and ground truth Y divided by the maximum length of the two strings: NED(Y,Ŷ) = Lev(Y,Ŷ) / max(|Y|,|Ŷ|). Lower values indicate higher similarity.
- `BLEU` — range: [0, 1]
  - Bilingual Evaluation Understudy. Calculates the geometric mean of n-gram precision, penalized for brevity using a Brevity Penalty (BP) based on generated and reference lengths.
- `ROUGE-L` — range: [0, 1]
  - Captures sentence-level structure via the Longest Common Subsequence (LCS). Computes the weighted F-measure of LCS precision and recall, where beta controls the precision-recall trade-off.
- `TEDS` — range: [0, 1]
  - Tree-Edit-Distance-based Similarity. Used for tables, it models content as trees (e.g., HTML DOM) and computes 1 - TED(T,Ť) / max(|T|,|Ť|). Higher scores indicate better structural and content reconstruction.

## Input / output format

**Input**: Images of shredded document fragments (8, 12, or 16 pieces) accompanied by a prompt instructing the model to reconstruct the original document content.

**Output**: Reconstructed text, code, or table in plain text or Markdown/HTML format.

## Scoring recipe

```python
def compute_metrics(pred, ref, is_table=False):
    ned = levenshtein_distance(pred, ref) / max(len(pred), len(ref))
    bleu = compute_bleu(pred, ref)
    rouge_l = compute_rouge_l_fscore(pred, ref)
    if is_table:
        teds = 1 - tree_edit_distance(pred, ref) / max(len(pred), len(ref))
        return ned, bleu, rouge_l, teds
    return ned, bleu, rouge_l
```

## Common pitfalls

- Chinese text scores are disproportionately penalized by BLEU and ROUGE due to the lack of explicit word delimiters, causing minor errors to disrupt segmentation boundaries.
- Python code reconstruction fails more frequently than Java or C++ because shredding disrupts whitespace-dependent indentation, whereas explicit delimiters in other languages act as visual anchors.
- Table reconstruction requires rigid 2D spatial alignment; models optimized for semantic flow may underperform on tabular data despite high semantic accuracy, as shown by divergences between NED and TEDS.

## Evidence (verbatim from paper)

> We employ Normalized Edit Distance (NED) for general text similarity. It normalizes the Levenshtein distance (Lev) between prediction Ŷ and ground truth Y: NED(Y,Ŷ) = Lev(Y,Ŷ) / max(|Y|,|Ŷ|). A lower NED implies higher similarity.

## Citation

```bibtex
@misc{guo2026shredbench,
  title={ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction},
  author={Guo et al. (2026)},
  year={2026},
  note={arXiv:2604.23813}
}
```

- arXiv: 2604.23813