Sketchduo Eval

---
name: sketchduo-eval
description: This evaluation probes a diffusion model's ability to generate pixel-level sketches that align with detailed text prompts while maintaining stylistic abstraction and human-like drawing characteristics. It measures both perceptual image quality and fine-grained text-to-image semantic alignment across multiple complementary metrics. Use when the user wants to benchmark on SketchDUO, or asks about evaluating this task. Reports TIFAScore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.20093
  bibtex_key: park2025stablesketcher
  confidence: high
---

# sketchduo-eval

> StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback — Park et al. (2025) (arXiv:2510.20093, 2025)

## What this evaluates

This evaluation probes a diffusion model's ability to generate pixel-level sketches that align with detailed text prompts while maintaining stylistic abstraction and human-like drawing characteristics. It measures both perceptual image quality and fine-grained text-to-image semantic alignment across multiple complementary metrics.

## Datasets

- **SketchDUO** — total 3000; splits: train (1800), test (1200)

## Metrics

- `FID` — range: [0, inf)
  - Fréchet Inception Distance measures the Wasserstein-2 distance between the distributions of real and generated sketch images in the Inception feature space. Lower values indicate better image quality and distribution matching.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity computes perceptual distance using deep network features. Lower values indicate higher perceptual similarity to reference sketches.
- `CLIPScore` — range: [0, 1]
  - Cosine similarity between image and text embeddings from the CLIP model. Higher values indicate stronger text-image alignment.
- `BERTScore` — range: [0, 1]
  - Precision, recall, and F1 computed over token-level embeddings from BERT, matching predictions to gold text via cosine similarity. Higher values indicate better semantic alignment.
- `TIFAScore` **(primary)** — range: [0, 1]
  - Text-Image Fine-grained Alignment Score evaluates element-level fidelity by parsing prompts into specific visual elements and measuring their presence/accuracy in generated images. Higher values indicate better prompt fidelity.

## Input / output format

**Input**: Text prompts describing sketch content, style, and background conditions (e.g., 'A black line drawing of a teddy bear with a friendly smile on a white background.').

**Output**: Pixel-based sketch images (binary/grayscale line drawings) generated by the diffusion model.

## Scoring recipe

```python
def evaluate_sketches(predictions, gold_prompts, real_sketches):
    results = {}
    results['FID'] = frechet_inception_distance(real_sketches, predictions)
    results['LPIPS'] = perceptual_l2_similarity(real_sketches, predictions)
    results['CLIPScore'] = clip_similarity(predictions, gold_prompts)
    results['BERTScore'] = bert_semantic_match(predictions, gold_prompts)
    results['TIFAScore'] = tifa_element_fidelity(predictions, gold_prompts)
    return results
```

## Common pitfalls

- Metric directionality confusion: FID and LPIPS are lower-is-better, whereas CLIPScore, BERTScore, and TIFAScore are higher-is-better.
- Over-reliance on BERTScore: BERTScore captures global semantic similarity but misses fine-grained instance details; TIFAScore is required for accurate prompt fidelity assessment in sketch generation.
- User study ranking interpretation: Lower mean rank indicates stronger preference, which contradicts standard accuracy benchmarks and can cause misinterpretation if not explicitly noted.

## Evidence (verbatim from paper)

> To evaluate the quality of the generated images, we adopt five metrics that encompass both image quality and text-image alignment. For image quality assessment, we employ Fréchet Inception Distance (FID) and LPIPS. For text-image alignment, we leverage CLIPScore, BERTScore, and TIFAScore.

## Citation

```bibtex
@misc{park2025stablesketcher,
  title={StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback},
  author={Park et al. (2025)},
  year={2025},
  note={arXiv:2510.20093}
}
```

- arXiv: 2510.20093