Sketch Rnn Eval

---
name: sketch-rnn-eval
description: Evaluates a generative model's ability to produce coherent stroke-based vector sketches through reconstruction, latent space interpolation, and completion of incomplete drawings. It probes how well the model captures conceptual features and organizes them in a continuous latent manifold. Use when the user wants to benchmark on QuickDraw, or asks about evaluating this task. Reports LR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1704.03477
  bibtex_key: ha2017sketchrnn
  confidence: high
---

# sketch-rnn-eval

> A Neural Representation of Sketch Drawings — Ha et al. (2017) (arXiv:1704.03477, 2017)

## What this evaluates

Evaluates a generative model's ability to produce coherent stroke-based vector sketches through reconstruction, latent space interpolation, and completion of incomplete drawings. It probes how well the model captures conceptual features and organizes them in a continuous latent manifold.

## Datasets

- **QuickDraw** — total ?; splits: test (-1)

## Metrics

- `LR` **(primary)** — range: other
  - Reconstruction loss, computed as the negative log-likelihood of the input stroke sequence given the latent code and previous strokes. Lower values indicate better reconstruction fidelity.
- `LK` — range: other
  - KL divergence between the approximate posterior distribution and the Gaussian prior over the latent space. Lower values indicate stronger regularization toward the prior.

## Input / output format

**Input**: A sequence of stroke coordinates/commands representing a human-drawn sketch.

**Output**: A sequence of stroke coordinates/commands representing a generated, reconstructed, or completed sketch.

## Scoring recipe

```python
def compute_metrics(predictions, gold, w_KL=1.0):
    # LR: Reconstruction loss (negative log-likelihood)
    LR = -sum(log_prob(gold[t] | gold[:t], latent) for t in range(len(gold)))
    # LK: KL divergence between encoder posterior and Gaussian prior
    LK = kl_divergence(encoder_posterior, prior)
    return {"LR": LR, "LK": LK}
```

## Common pitfalls

- Qualitative assessments (interpolation, analogies, completion) are subjective and lack quantitative metrics.
- Loss values are highly class-dependent due to varying sketch complexity, making cross-class comparisons misleading.
- Results are reported for multiple w_KL hyperparameters; comparing across settings without specifying w_KL is invalid.

## Evidence (verbatim from paper)

> The results for test set evaluation on various datasets are displayed in Table 1. We see that the reconstruction loss term $L_{R}$ decreases as we relax the $w_{KL}$ parameter controlling the weight for the KL loss term, and meanwhile the KL loss term $L_{R}$ increases as a result.

## Citation

```bibtex
@misc{ha2017sketchrnn,
  title={A Neural Representation of Sketch Drawings},
  author={Ha et al. (2017)},
  year={2017},
  note={arXiv:1704.03477}
}
```

- arXiv: 1704.03477