Postercraft Text Eval

---
name: postercraft-text-eval
description: Evaluates the ability of text-to-image models to accurately render specified textual elements within aesthetically designed posters. It measures how well generated images preserve the exact characters, words, and layout instructions from the input prompt. Use when the user wants to benchmark on PosterCraft Test Prompts, or asks about evaluating this task. Reports Text F-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.10741
  bibtex_key: chen2025postercraft
  confidence: high
---

# postercraft-text-eval

> PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Framework — Sixiang Chen et al. (2025) (arXiv:2506.10741, 2025)

## What this evaluates

Evaluates the ability of text-to-image models to accurately render specified textual elements within aesthetically designed posters. It measures how well generated images preserve the exact characters, words, and layout instructions from the input prompt.

## Datasets

- **PosterCraft Test Prompts** — total 100; splits: test (100)

## Metrics

- `Text Recall` — range: [0, 1]
  - Measures the proportion of ground-truth prompt text characters or words successfully recognized by the OCR engine in the generated image.
- `Text F-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for text recognition, balancing exact character/word matching against coverage of the prompt text.
- `Text Accuracy` — range: [0, 1]
  - Proportion of generated images where the OCR-extracted text exactly matches the ground-truth prompt text without errors.

## Input / output format

**Input**: Natural language prompts describing aesthetic poster content, style, and specific textual elements to be rendered.

**Output**: High-resolution poster images containing the specified text and visual design elements.

## Scoring recipe

```python
prompts = load_100_prompts()
images = [generate(prompt) for prompt in prompts]  # 3 samples per prompt
extracted_texts = [ocr_engine(img) for img in images]
metrics = compute_ocr_metrics(extracted_texts, prompts)
final_recall = mean(metrics['recall'])
final_fscore = mean(metrics['f1'])
final_accuracy = mean(metrics['accuracy'])
```

## Common pitfalls

- OCR engines frequently fail on highly stylized, rotated, or artistically distorted text common in aesthetic posters.
- Evaluating exact character match requires careful normalization of case, punctuation, and spacing between the prompt and rendered text.
- Short vs. long prompt lengths may require different evaluation thresholds for text legibility and layout adherence.

## Evidence (verbatim from paper)

> To assemble our test prompts, we use Gemini2.0-Flash-Gen*(Team et al., [2023])* to randomly generate 100 aesthetic poster prompts—balanced across short, medium, and long lengths—and for each prompt we sample three output per model, yielding 300 test images. We generate posters with OpenCOLE*(Inoue et al., [2024])*, Playground-v2.5*(Liu et al., [2024a])*, SD3.5*(AI, [2024])*, Flux1.dev*(https://github.com/black-forest labs/flux, [2024])*, Ideogram-v2*(v2. https://ideogram.ai/launch, [2024])*, BAGEL*(Deng et al., [2025])*, and Gemini2.0-Flash-Gen*(Team et al., [2023])*, apply the OCR engine of the SOTA VLM*(Team et al., [2023])* to each image, and report three precision-oriented metrics—text recall, text F-score, and text accuracy—averaged across all 300 samples.

## Citation

```bibtex
@misc{chen2025postercraft,
  title={PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Framework},
  author={Sixiang Chen et al. (2025)},
  year={2025},
  note={arXiv:2506.10741}
}
```

- arXiv: 2506.10741