Interaction2code Eval

---
name: interaction2code-eval
description: Evaluates multimodal large language models' ability to generate functional interactive webpage code from interactive prototype screenshots. It specifically probes the model's capacity to capture dynamic interaction behaviors, element positioning, and visual-textual alignment, rather than just static layout reproduction. Use when the user wants to benchmark on Interaction2Code, or asks about evaluating this task. Reports CLIP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.03292
  bibtex_key: [REDACTED]
  confidence: high
---

# interaction2code-eval

> Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping — Jingyu Xiao et al. (2024) (arXiv:2411.03292, 2024)

## What this evaluates

Evaluates multimodal large language models' ability to generate functional interactive webpage code from interactive prototype screenshots. It specifically probes the model's capacity to capture dynamic interaction behaviors, element positioning, and visual-textual alignment, rather than just static layout reproduction.

## Datasets

- **Interaction2Code** — total ?; splits: train (-1), test (-1); repo https://github.com/WebPAI/Interaction2Code

## Metrics

- `CLIP` **(primary)** — range: [0, 1]
  - Cosine similarity between CLIP image embeddings of the rendered generated webpage and the reference webpage screenshot.
- `SSIM` — range: [0, 1]
  - Structural Similarity Index Measure comparing the pixel-level structural fidelity between the generated and reference webpage screenshots.
- `Text` — range: [0, 1]
  - Textual overlap or embedding similarity between the extracted text content of the generated code and the reference code.
- `Position` — range: [0, 1]
  - Positional similarity score measuring the alignment of interactive elements' coordinates between the generated and reference webpages.
- `IR` — range: [0, 1]
  - Interaction Rate, representing the proportion of correctly generated interactive elements relative to the ground truth.
- `Usability Rate` — range: [0, 1]
  - Percentage of generated interactions deemed functionally consistent with the ground truth by human annotators with front-end development experience.
- `Win Rate` — range: [0, 1]
  - Proportion of pairwise comparisons where the model's output is preferred over the baseline, determined by majority vote (≥3 out of 5 annotators).

## Input / output format

**Input**: Screenshot of an interactive webpage prototype (and optionally accompanying textual descriptions).

**Output**: HTML/CSS/JavaScript code implementing the interactive webpage.

## Scoring recipe

```python
def compute_metrics(generated_code, ref_code, gen_img, ref_img):
    gen_img, ref_img = render_to_image(generated_code), render_to_image(ref_code)
    clip = cosine_similarity(CLIP.encode(gen_img), CLIP.encode(ref_img))
    ssim = structural_similarity(gen_img, ref_img)
    text = text_similarity(extract_text(generated_code), extract_text(ref_code))
    pos = element_position_similarity(extract_elements(generated_code), extract_elements(ref_code))
    ir = count_correct_interactions(generated_code, ref_code) / count_total_interactions(ref_code)
    return {'CLIP': clip, 'SSIM': ssim, 'Text': text, 'Position': pos, 'IR': ir}
```

## Common pitfalls

- MLLMs significantly undergenerate interactive elements relative to static content, leading to incomplete functionality.
- Performance degrades sharply for interactions with low visual saliency (small area ratio on the screen).
- Single-modality (visual-only) inputs yield substantially poorer results than combined visual-textual inputs.

## Evidence (verbatim from paper)

> Gemini-1.5-flash's metrics (CLIP, SSIM, text, position, IR) of the interaction part improve from direct prompting scores (0.4737, 0.3616, 0.2809, 0.4302, 0.6738) to (0.5093, 0.3854, 0.3217, 0.4511, 0.7112) with CoT, and further to (0.5194, 0.3898, 0.3454, 0.4612, 0.7326) with Mark prompting.

## Citation

```bibtex
@misc{xiao2024interaction2code,
  title={Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping},
  author={Jingyu Xiao et al. (2024)},
  year={2024},
  note={arXiv:2411.03292}
}
```

- arXiv: 2411.03292