Vision2web Eval

---
name: vision2web-eval
description: Evaluates multimodal coding agents' ability to generate, deploy, and verify visual website development tasks across three hierarchical levels: static webpage generation, interactive frontend development, and full-stack website construction. It probes long-horizon planning, cross-modal reasoning, multi-page coordination, and visual/functional fidelity. Use when the user wants to benchmark on Vision2Web, or asks about evaluating this task. Reports Visual Score (VS), Functional Score (FS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.26648
  bibtex_key: he2026vision2web
  confidence: high
---

# vision2web-eval

> Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification — He et al. (2026) (arXiv:2603.26648, 2026)

## What this evaluates

Evaluates multimodal coding agents' ability to generate, deploy, and verify visual website development tasks across three hierarchical levels: static webpage generation, interactive frontend development, and full-stack website construction. It probes long-horizon planning, cross-modal reasoning, multi-page coordination, and visual/functional fidelity.

## Datasets

- **Vision2Web** — total ?; splits: test (-1)

## Metrics

- `Visual Score (VS)` **(primary)** — range: [0, 100]
  - A VLM-based judge (Gemini-3-Pro-Preview) scores the visual fidelity of the deployed website against the prototype image on a 0–100 scale.
- `Functional Score (FS)` **(primary)** — range: [0, 100]
  - A GUI agent verifier (GLM-4.6V) executes predefined test workflows on the deployed site. The score is the pass rate/functional correctness normalized to a 0–100 scale.
- `Deployment Success Rate (DSR)` — range: percent
  - The percentage of tasks that successfully generate a startup script, deploy on a fixed port, and run without errors within a 10-minute timeout.

## Input / output format

**Input**: Prototype images, textual requirements, and multimedia resources provided in a working directory, along with carefully designed prompts guiding the required level of project completion.

**Output**: Generated project files, a startup script to run the project on a fixed port, and a successfully deployed web application.

## Scoring recipe

```python
def score_task(agent_output, gold_spec):
    # 1. Deployment check
    if not deploy(agent_output, timeout=600):
        return {"DSR": 0, "FS": 0, "VS": 0}
    # 2. Functional Score via GUI Agent Verifier
    test_results = gui_agent_verify(agent_output, gold_spec.workflows)
    FS = (sum(test_results.passed) / len(test_results)) * 100
    # 3. Visual Score via VLM Judge
    VS = vlm_judge_score(agent_output.screenshot, gold_spec.prototype)
    # 4. Deployment Success Rate
    DSR = 1.0
    return {"DSR": DSR, "FS": FS, "VS": VS}
```

## Common pitfalls

- Deployment timeout (10 min) or errors automatically zero out all scores, which can disproportionately penalize agents that take longer to compile or run.
- Visual and Functional scores are evaluated by different specialized models (VLM judge vs GUI agent), so results are not purely end-to-end model capabilities but depend on the verifier setup.
- DSR is explicitly noted as 'for reference rather than an official metric' and should not be conflated with the primary VS/FS scores.

## Evidence (verbatim from paper)

> Table 3: End-to-end performance of multimodal coding agents on Vision2Web across three task levels, reporting device-wise static scores, averaged functional scores (FS) and visual scores (VS) for interactive and full-stack tasks, with Deployment Success Rate (DSR) provided for reference rather than an official metric.

## Citation

```bibtex
@misc{he2026vision2web,
  title={Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2603.26648}
}
```

- arXiv: 2603.26648