Kvbench Eval

---
name: kvbench-eval
description: Evaluates text-to-image models on knowledge-intensive generation across six high-school academic subjects and two languages. It probes scientific fidelity, logical reasoning, symbolic precision, and multilingual robustness using textbook-derived prompts and atomic checklist verification. Use when the user wants to benchmark on KVBench, or asks about evaluating this task. Reports performance_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.22302
  bibtex_key: zhao2026knowledgevisualization
  confidence: medium
---

# kvbench-eval

> Knowledge Visualization: A Benchmark and Method for Knowledge-Intensive Text-to-Image Generation — Ran Zhao et al. (2026) (arXiv:2604.22302, 2026)

## What this evaluates

Evaluates text-to-image models on knowledge-intensive generation across six high-school academic subjects and two languages. It probes scientific fidelity, logical reasoning, symbolic precision, and multilingual robustness using textbook-derived prompts and atomic checklist verification.

## Datasets

- **KVBench** — total ?; splits: test (-1); repo https://github.com/zhaoran66/KVBench

## Metrics

- `performance_score` **(primary)** — range: percent
  - Percentage of generated images that pass an atomic checklist of domain-specific knowledge constraints and visual elements. Calculated as (number of passing instances / total instances) * 100.

## Input / output format

**Input**: Text prompts (Brief Caption or Detailed Caption) describing academic concepts, provided in either Chinese or English.

**Output**: A single generated image per prompt.

## Scoring recipe

```python
def compute_performance_score(predictions, gold_checklists):
    passed = 0
    for pred_img, checklist in zip(predictions, gold_checklists):
        if all(check_constraint(pred_img, c) for c in checklist):
            passed += 1
    return (passed / len(predictions)) * 100
```

## Common pitfalls

- Detailed captions can degrade performance for weaker open-source models due to poor instruction-following and interference.
- Models show significant multilingual fragility, with English prompts consistently outperforming Chinese ones.
- Performance gaps are largest in reasoning-heavy subjects (Math, Physics) requiring symbolic precision and multi-step logic.

## Evidence (verbatim from paper)

> Table 2 presents a comprehensive comparison of model performance across six academic disciplines and two languages (“zh” for Chinese and “en” for English.

## Citation

```bibtex
@misc{zhao2026knowledgevisualization,
  title={Knowledge Visualization: A Benchmark and Method for Knowledge-Intensive Text-to-Image Generation},
  author={Ran Zhao et al. (2026)},
  year={2026},
  note={arXiv:2604.22302}
}
```

- arXiv: 2604.22302