Graphic Design Bench Eval

---
name: graphic-design-bench-eval
description: Evaluates AI systems' ability to perceive, reason about, and generate professional graphic design artifacts across layout, typography, vector graphics, template semantics, and animation. It probes multi-constraint satisfaction through design-native metrics measuring spatial accuracy, perceptual quality, and semantic alignment. Use when the user wants to benchmark on LICA layered-composition dataset, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.04192
  bibtex_key: deganutti2026graphicdesignbench
  confidence: high
---

# graphic-design-bench-eval

> Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks — Deganutti et al. (2026) (arXiv:2604.04192, 2026)

## What this evaluates

Evaluates AI systems' ability to perceive, reason about, and generate professional graphic design artifacts across layout, typography, vector graphics, template semantics, and animation. It probes multi-constraint satisfaction through design-native metrics measuring spatial accuracy, perceptual quality, and semantic alignment.

## Datasets

- **LICA layered-composition dataset** — total ?; splits: test (-1); repo https://github.com/purvanshi-lica/lica-bench

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection over Union; measures average overlap between predicted and ground-truth bounding boxes.
- `CLIP Score` — range: [0, 1]
  - Cosine similarity between image and text embeddings, measuring semantic alignment.
- `LPIPS` — range: [0, 1]
  - Learned perceptual distance between two images using deep features; lower values indicate greater perceptual similarity.
- `SSIM` — range: [0, 1]
  - Structural similarity index based on luminance, contrast, and structure; higher is better.
- `FID` — range: [0, 1]
  - Fréchet Inception Distance; measures distributional distance between generated and reference image sets in Inception feature space.

## Input / output format

**Input**: Varies by task: text-only (layout JSON or text metadata), image-only (rendered PNG), or both. For video tasks, native video or uniformly sampled keyframes. All evaluated with greedy decoding (temperature=0) using fixed prompt templates.

**Output**: Task-dependent: natural language/text responses for understanding/classification, rendered images, SVG code, Lottie JSON, or video files for generation tasks.

## Scoring recipe

```python
def score(predictions, gold):
    # Spatial accuracy (mIoU)
    ious = [intersection_over_union(pred_box, gold_box) for pred_box, gold_box in zip(predictions, gold)]
    mIoU = sum(ious) / len(ious)
    # Semantic alignment (CLIP Score)
    clip_scores = [cosine_similarity(clip_encode(pred), clip_encode(gold)) for pred, gold in zip(predictions, gold)]
    # Perceptual quality (LPIPS)
    lpips_scores = [lpips_metric(pred_img, gold_img) for pred_img, gold_img in zip(predictions, gold)]
    return {"mIoU": mIoU, "CLIP_Score": mean(clip_scores), "LPIPS": mean(lpips_scores)}
```

## Common pitfalls

- Modality mismatch: Not all models support native video or specific input types; keyframe extraction is used as a fallback, which may skew temporal reasoning results.
- Sample size disparity: Understanding tasks use ~1,000-2,500 samples, while generation tasks are limited to ~100 due to API costs, affecting statistical power.
- Expert-defined solvability thresholds: Performance is categorized as 'Mostly Solved' (>95%), 'Partially Solved' (80-95%), or 'Unsolved' (<80%) based on design expert consultation, not standard academic baselines.

## Evidence (verbatim from paper)

> Throughout the paper we assign each task one of three solvability labels. Mostly Solved: best-model performance exceeds 95% (or an equivalent metric-specific threshold), with limited room for improvement at the current evaluation granularity. Partially Solved: best-model performance falls between 80–95%... mIoU (mean Intersection over Union) measures average overlap between predicted and ground-truth bounding boxes. CLIP Score computes cosine similarity between image and text embeddings. LPIPS computes learned perceptual distance between two images using deep features; lower values indicate greater perceptual similarity.

## Citation

```bibtex
@misc{deganutti2026graphicdesignbench,
  title={Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks},
  author={Deganutti et al. (2026)},
  year={2026},
  note={arXiv:2604.04192}
}
```

- arXiv: 2604.04192