Webgen Bench Eval

---
name: webgen-bench-eval
description: Evaluates a model's ability to generate functional and visually accurate website codebases from natural language instructions. It measures both functional correctness via automated GUI-agent testing and visual fidelity via VLM-based appearance scoring. Use when the user wants to benchmark on WebGen-Bench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22644
  bibtex_key: lu2025webgenagent
  confidence: high
---

# webgen-bench-eval

> WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning — Lu et al. (2025) (arXiv:2509.22644, 2025)

## What this evaluates

Evaluates a model's ability to generate functional and visually accurate website codebases from natural language instructions. It measures both functional correctness via automated GUI-agent testing and visual fidelity via VLM-based appearance scoring.

## Datasets

- **WebGen-Bench** — total 101; splits: test (101); repo https://github.com/mnluzimu/WebGen-Agent

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of instructions where the generated website passes functional tests. Calculated as (count(Yes) + 0.5 * count(Partial)) / total_instructions * 100, where 'Yes' means all 647 GUI-agent tests pass, 'Partial' means some pass, and 'No' means none pass.
- `Appearance Score` — range: [1, 5]
  - Average visual quality score assigned by a VLM (GPT-4o or Qwen2.5-VL-32B-Instruct) to screenshots of the rendered website. Scores range from 1 to 5.

## Input / output format

**Input**: Natural language instruction describing the desired website functionality and layout.

**Output**: Complete website codebase (HTML, CSS, JavaScript files) ready for rendering.

## Scoring recipe

```python
def compute_metrics(predictions, gold_instructions):
    total = len(gold_instructions)
    yes_count = 0
    partial_count = 0
    for inst, code in zip(gold_instructions, predictions):
        test_results = run_gui_agent_tests(code, num_tests=647)
        if all(test_results):
            yes_count += 1
        elif any(test_results):
            partial_count += 1
    accuracy = (yes_count + 0.5 * partial_count) / total * 100
    appearance_scores = [vlm_score(screenshot(code)) for code in predictions]
    appearance_score = sum(appearance_scores) / len(appearance_scores)
    return accuracy, appearance_score
```

## Common pitfalls

- Accuracy weights partial test passes at 0.5, which may undercount functionality compared to a strict pass/fail metric.
- Appearance Score is highly sensitive to the VLM used for evaluation; switching from GPT-4o to Qwen2.5-VL-32B-Instruct can shift scores by ~0.2 points.
- Start Failed cases are excluded from functional scoring but indicate environment or setup issues that should be reported separately.

## Evidence (verbatim from paper)

> We evaluate WebGen-Agent using WebGen-Bench, a benchmark containing 101 website-generation instructions in natural language and 647 GUI-agent test cases, covering a wide range of web applications. Following Lu et al. (2025b), we use Qwen2.5-VL-32B-Instruct in functional testing and GPT-4o in appearance evaluation. The highest Accuracy and Appearance Score are highlighted in bold.

## Citation

```bibtex
@misc{lu2025webgenagent,
  title={WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2509.22644}
}
```

- arXiv: 2509.22644