Image2struct Eval

---
name: image2struct-eval
description: Evaluates vision-language models' ability to extract structural code (HTML, LaTeX, LilyPond) from images. It uses a round-trip validation pipeline where generated code is rendered back to an image and compared to the original using automated similarity metrics. Use when the user wants to benchmark on Image2Struct, or asks about evaluating this task. Reports EMS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.22456
  bibtex_key: somerville2024image2struct
  confidence: high
---

# image2struct-eval

> Image2Struct: Benchmarking Structure Extraction for Vision-Language Models — Somerville Roberts et al. (2024) (arXiv:2410.22456, 2024)

## What this evaluates

Evaluates vision-language models' ability to extract structural code (HTML, LaTeX, LilyPond) from images. It uses a round-trip validation pipeline where generated code is rendered back to an image and compared to the original using automated similarity metrics.

## Datasets

- **Image2Struct** — total ?; splits: test (-1); repo https://github.com/stanford-crfm/image2struct

## Metrics

- `EMS` **(primary)** — range: [0, 1]
  - Automated image similarity metric comparing the original input image to the rendered output image, designed to correlate strongly with structural edit distances.
- `compilation success rate` — range: [0, 1]
  - Fraction of generated code snippets that successfully compile and render into an image without errors.

## Input / output format

**Input**: An image of a structure (webpage, LaTeX document, or musical score) paired with a zero-shot prompt specifying the expected structural code format.

**Output**: The structural code (HTML, LaTeX, or LilyPond) intended to reproduce the input image.

## Scoring recipe

```python
compilation_success = 0
ems_scores = []
for pred_code, gold_img in zip(predictions, gold_images):
    rendered_img = render_and_screenshot(pred_code)
    if rendered_img is not None:
        compilation_success += 1
        ems_scores.append(earth_mover_similarity(gold_img, rendered_img))
    else:
        ems_scores.append(0.0)
return {
    'compilation_success_rate': compilation_success / len(predictions),
    'earth_mover_similarity': sum(ems_scores) / len(ems_scores)
}
```

## Common pitfalls

- Models frequently refuse to generate code for musical scores due to false copyright infringement flags.
- Generated code often contains syntax errors or missing packages, requiring automated post-processing before rendering.
- Evaluation runs only once per model per instance (temperature=0) to minimize cost, potentially overlooking stochastic response variability.

## Evidence (verbatim from paper)

> We consider rendering success rate as a metric for model performance. ... We rank models with the mean win rate—which is the average fraction of other models that a model outperforms across scenarios—using the compilation success rates and EMS scores. Any compilation failure counts as zero EMS in the mean win rate calculation.

## Citation

```bibtex
@misc{somerville2024image2struct,
  title={Image2Struct: Benchmarking Structure Extraction for Vision-Language Models},
  author={Somerville Roberts et al. (2024)},
  year={2024},
  note={arXiv:2410.22456}
}
```

- arXiv: 2410.22456