Frontalk Eval

---
name: frontalk-eval
description: Probes a model's ability to generate and iteratively refine front-end code through multi-turn conversational instructions, handling both textual and visual feedback. It specifically measures functional correctness, user experience design quality, and the model's tendency to overwrite prior implementations in long-context interactions. Use when the user wants to benchmark on FronTalk, or asks about evaluating this task. Reports pass rate (PR), usability (UX).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.04203
  bibtex_key: wu2025frontalk
  confidence: high
---

# frontalk-eval

> FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback — Wu et al. (2025) (arXiv:2601.04203, 2025)

## What this evaluates

Probes a model's ability to generate and iteratively refine front-end code through multi-turn conversational instructions, handling both textual and visual feedback. It specifically measures functional correctness, user experience design quality, and the model's tendency to overwrite prior implementations in long-context interactions.

## Datasets

- **FronTalk** — total ?; splits: test (-1); repo https://github.com/shirley-wu/frontalk

## Metrics

- `pass rate (PR)` **(primary)** — range: percent
  - Percentage of generated code instances that successfully pass functional and visual checks verified by an automated GPT-4o-powered web agent.
- `usability (UX)` **(primary)** — range: percent
  - Score reflecting the quality of user interaction trajectories simulated by a web agent, evaluated against human-aligned automated judges based on design principles and functionality.
- `forgetting rate (FR)` — range: percent
  - Proportion of multi-turn tasks where the model overwrites or loses previously implemented functionality when processing new instructions for the same code component.

## Input / output format

**Input**: A sequence of multi-turn user instructions (textual or multi-modal with UI screenshots/annotations) specifying front-end development tasks and iterative modifications.

**Output**: Generated front-end code (e.g., HTML/CSS/JavaScript) for each turn in the conversation.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    agent = WebAgent(model='GPT-4o')
    pr, ux, fr = [], [], []
    for pred in predictions:
        result = agent.run_code(pred.code)
        pr.append(1.0 if result.passes_functional_and_visual_checks() else 0.0)
        ux.append(agent.simulate_usability_trajectory(pred.code))
        if pred.is_multi_turn:
            fr.append(1.0 if result.overwrites_prior_functionality() else 0.0)
    return {
        'PR': sum(pr) / len(pr),
        'UX': sum(ux) / len(ux),
        'FR': sum(fr) / len(fr)
    }
```

## Common pitfalls

- Models frequently overwrite previous code implementations instead of integrating new features, leading to high forgetting rates in multi-turn settings.
- Visual feedback interpretation is significantly harder than textual, with open-source VLMs often missing implicit functionalities or dense text annotations.
- Long-context handling degrades performance in later turns for some models, independent of the actual task difficulty.

## Evidence (verbatim from paper)

> For evaluation, we employ the web agent powered by GPT-4o to perform agent-based evaluation. We report pass rate (PR) and usability (UX) as main metrics and forgetting rate (FR) as a supplementary metric to measure the forgetting issue.

## Citation

```bibtex
@misc{wu2025frontalk,
  title={FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2601.04203}
}
```

- arXiv: 2601.04203