Weavebench Eval

---
name: weavebench-eval
description: Evaluates multi-turn, context-aware image comprehension and generation in an interleaved setting. It probes a model's ability to maintain visual consistency, follow iterative editing instructions, and integrate historical context across multiple turns. Use when the user wants to benchmark on WEAVEBench, or asks about evaluating this task. Reports WEAVEBench.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.11434
  bibtex_key: chow2025weave
  confidence: high
---

# weavebench-eval

> WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation — Chow et al. (2025) (arXiv:2511.11434, 2025)

## What this evaluates

Evaluates multi-turn, context-aware image comprehension and generation in an interleaved setting. It probes a model's ability to maintain visual consistency, follow iterative editing instructions, and integrate historical context across multiple turns.

## Datasets

- **WEAVEBench** — total ?; splits: test (-1)

## Metrics

- `WEAVEBench` **(primary)** — range: [0, 1]
  - Average score across four categories: ΔScience, Creation, Logic, and Game. Scores are computed via a hybrid VLM judge (GPT-4.1) and normalized to [0, 1].

## Input / output format

**Input**: Multi-turn dialogue containing interleaved text instructions and images. Models are evaluated under three in-context conditions: (1) no in-context (single-turn), (2) partial in-context (only self-generated images with explicit visual context), and (3) complete in-context (all previous interactions visible). Images can be placed at first mention ('yes-first') or consolidated at the beginning ('yes-front').

**Output**: Generated image(s) corresponding to the final instruction in the dialogue, or text response for comprehension tasks.

## Scoring recipe

```python
def compute_weavebench_score(predictions, gold, judge='GPT-4.1'):
    scores = []
    for pred, gold_inst in zip(predictions, gold):
        score = judge.evaluate(pred, gold_inst)
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Confusing the three in-context conditions (no, partial, complete) which significantly impact performance, especially for open-source vs proprietary models.
- Image placement strategy ('yes-first' vs 'yes-front') affects results; concatenation can cause up to 10.3% degradation compared to sequential input.
- Relying solely on automated VLM judges without verifying human correlation, though the paper shows >0.8 Pearson correlation.

## Evidence (verbatim from paper)

> We evaluated 4 LLMs, 7 Edit models, and 11 UMMs on WEAVEBench as presented in Table 2. Evaluations were conducted under three distinct in-context conditions: (1) no in-context (single-turn generation without contextual information), (2) partial in-context (using only self-generated images with explicitly mentioned visual context, excluding other historical interactions), and (3) complete in-context (with all previous interactions visible).

## Citation

```bibtex
@misc{chow2025weave,
  title={WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation},
  author={Chow et al. (2025)},
  year={2025},
  note={arXiv:2511.11434}
}
```

- arXiv: 2511.11434