Layerd Eval

---
name: layerd-eval
description: Evaluates a model's ability to decompose raster graphic designs into a sequence of re-editable layers. It measures visual reconstruction quality and the number of edits required to match a ground-truth layer structure, accounting for the ill-posed nature of layer ordering. Use when the user wants to benchmark on Crello, or asks about evaluating this task. Reports RGB L1, Alpha IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.25134
  bibtex_key: suzuki2025layerd
  confidence: high
---

# layerd-eval

> LayerD: Decomposing Raster Graphic Designs into Layers — Suzuki et al. (2025) (arXiv:2509.25134, 2025)

## What this evaluates

Evaluates a model's ability to decompose raster graphic designs into a sequence of re-editable layers. It measures visual reconstruction quality and the number of edits required to match a ground-truth layer structure, accounting for the ill-posed nature of layer ordering.

## Datasets

- **Crello** — total ?; splits: train (19478), val (1852), test (1971)

## Metrics

- `RGB L1` **(primary)** — range: [0, 255]
  - Mean absolute error between predicted and ground-truth pixel RGB values across the image.
- `Alpha IoU` **(primary)** — range: [0, 1]
  - Intersection over Union of predicted and ground-truth alpha (transparency) masks.
- `Edit Distance (DTW)` — range: integer
  - Number of edits (insertions/deletions) to align predicted layer sequence with ground truth, computed using Dynamic Time Warping to handle ambiguous layer ordering.

## Input / output format

**Input**: Raster graphic design image (aspect ratio preserved, shorter side resized to 512 pixels).

**Output**: Ordered sequence of layers, each containing a segmentation mask, alpha channel, color map, and z-index (depth order).

## Scoring recipe

```python
def compute_metrics(pred_layers, gt_layers):
    # Align sequences using DTW to handle ordering ambiguity
    aligned_pred, aligned_gt = dtw_align(pred_layers, gt_layers)
    rgb_l1 = 0
    alpha_iou = 0
    for p, g in zip(aligned_pred, aligned_gt):
        rgb_l1 += np.mean(np.abs(p.rgb - g.rgb))
        alpha_iou += jaccard_index(p.alpha, g.alpha)
    return rgb_l1 / len(aligned_pred), alpha_iou / len(aligned_pred)
```

## Common pitfalls

- Transparent layers are explicitly excluded from evaluation.
- Text layers are evaluated separately from non-text layers.
- Edit distance uses Dynamic Time Warping (DTW) to handle the ill-posed, ambiguous nature of ground-truth layer structures.
- Metrics are reported as a function of the maximum number of allowed edits (0 to 5).

## Evidence (verbatim from paper)

> We show visual quality metrics (RGB L1, Alpha IoU) as the maximum number of allowed edits increases.

## Citation

```bibtex
@misc{suzuki2025layerd,
  title={LayerD: Decomposing Raster Graphic Designs into Layers},
  author={Suzuki et al. (2025)},
  year={2025},
  note={arXiv:2509.25134}
}
```

- arXiv: 2509.25134