Webcode2m Eval

---
name: webcode2m-eval
description: This benchmark evaluates multimodal models' ability to translate webpage design screenshots into functional HTML/CSS code. It probes visual fidelity, structural hierarchy recall, and the capacity to generate long, complex, real-world front-end code from visual inputs. Use when the user wants to benchmark on WebCode2M, or asks about evaluating this task. Reports TreeBLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.06369
  bibtex_key: gui2024webcode2m
  confidence: high
---

# webcode2m-eval

> WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs — Gui et al. (2024) (arXiv:2404.06369, 2024)

## What this evaluates

This benchmark evaluates multimodal models' ability to translate webpage design screenshots into functional HTML/CSS code. It probes visual fidelity, structural hierarchy recall, and the capacity to generate long, complex, real-world front-end code from visual inputs.

## Datasets

- **WebCode2M** — total 2560000; splits: train (2560000), WebCode2M-Short (256), WebCode2M-Mid (256), WebCode2M-Long (256)

## Metrics

- `CLIP similarity` — range: [0, 1]
  - Cosine similarity between the latent vectors of the reference design image and the rendered image of the generated code, both encoded by CLIP.
- `Visual Score` — range: [0, 1]
  - Average matching ratio between reference and candidate blocks, combined with similarity scores across four block levels: color, text, CLIP, and position.
- `TreeBLEU` **(primary)** — range: [0, 1]
  - Proportion of 1-height subtrees in the generated DOM tree that match the reference tree. Formulated as |S(t) ∩ S(ŝ)| / |S(ŝ)|, where S(·) is the set of 1-height subtrees, t is the reference tree, and ŝ is the generated tree. Terminal nodes and tag attributes are excluded.

## Input / output format

**Input**: A screenshot/image of a webpage design, optionally accompanied by a text prompt (e.g., "write an HTML code").

**Output**: Raw HTML/CSS code representing the webpage design.

## Scoring recipe

```python
def compute_metrics(gold_code, pred_code, image):
    # CLIP Similarity
    img_emb = clip.encode(image)
    rendered_img = render_html_css(pred_code)
    pred_emb = clip.encode(rendered_img)
    clip_sim = cosine_similarity(img_emb, pred_emb)

    # Visual Score
    blocks_ref = extract_blocks(gold_code)
    blocks_pred = extract_blocks(pred_code)
    visual_score = avg(match_ratio(blocks_ref, blocks_pred),
                       level_sim(blocks_ref, blocks_pred, ['color','text','clip','pos']))

    # TreeBLEU
    dom_gold = parse_dom(gold_code, remove_attributes=True)
    dom_pred = parse_dom(pred_code, remove_attributes=True)
    subtrees_gold = get_1_height_subtrees(dom_gold)
    subtrees_pred = get_1_height_subtrees(dom_pred)
    treebleu = len(subtrees_gold & subtrees_pred) / len(subtrees_pred)
    return clip_sim, visual_score, treebleu
```

## Common pitfalls

- TreeBLEU explicitly excludes terminal nodes and tag attributes (e.g., content, style) when extracting 1-height subtrees; including them will inflate scores incorrectly.
- All baselines must use a one-pass generation strategy to ensure fair comparison with specialized models fine-tuned on single-turn datasets.
- Models must handle extreme aspect ratios and variable sequence lengths without degradation, as the base architecture is specifically noted for this robustness.

## Evidence (verbatim from paper)

> TreeBLEU is defined as the proportion of all 1-height subtrees (see Algorithm 1) in a given tree that can be matched with that of a reference tree. Let S(·) be the set of 1-height subtrees, then it can be formulated as: TreeBLEU = |S(t) ∩ S(ŝ)| / |S(ŝ)|, where t and ŝ denote the given and reference trees, respectively.

## Citation

```bibtex
@misc{gui2024webcode2m,
  title={WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs},
  author={Gui et al. (2024)},
  year={2024},
  note={arXiv:2404.06369}
}
```

- arXiv: 2404.06369