Svg Sophia Refinement Eval

---
name: svg-sophia-refinement-eval
description: Evaluates a model's ability to refine and correct imperfect SVG code, measuring structural accuracy, visual fidelity, and code efficiency. Use when the user wants to benchmark on SVG-Sophia Code Refinement Benchmark, or asks about evaluating this task. Reports SR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.16189
  bibtex_key: wang2026ctrls
  confidence: high
---

# svg-sophia-refinement-eval

> Reliable Reasoning in SVG-LLMs via Multi-Task Multi-Reward Reinforcement Learning — Haomin Wang et al. (2026) (arXiv:2603.16189, 2026)

## What this evaluates

Evaluates a model's ability to refine and correct imperfect SVG code, measuring structural accuracy, visual fidelity, and code efficiency.

## Datasets

- **SVG-Sophia Code Refinement Benchmark** — total ?; splits: test (-1)

## Metrics

- `SR` **(primary)** — range: percent
  - Success Rate: percentage of refined SVGs that successfully render and match the target structure.
- `DINO` — range: [0, 1]
  - DINO similarity score between the refined SVG image and the ground truth.
- `SSIM` — range: [0, 1]
  - Structural Similarity Index measuring perceptual similarity between images.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity measuring perceptual distance between images.
- `Tokens` — range: other
  - Number of tokens in the refined SVG code, measuring code efficiency.

## Input / output format

**Input**: Imperfect SVG code draft.

**Output**: Refined SVG code string.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    rendered_imgs = [render_svg(svg) for svg in predictions]
    sr = sum(1 for img in rendered_imgs if img.is_valid()) / len(predictions)
    dino = dino_similarity(gold_imgs, rendered_imgs)
    ssim = structural_similarity(gold_imgs, rendered_imgs)
    lpips = perceptual_distance(gold_imgs, rendered_imgs)
    tokens = [count_tokens(svg) for svg in predictions]
    return {'SR': sr, 'DINO': dino, 'SSIM': ssim, 'LPIPS': lpips, 'Tokens': tokens}
```

## Common pitfalls

- Metrics compare the rendered output of the refined SVG against the ground truth image.
- Token count measures the length of the generated SVG code, not the number of characters.

## Evidence (verbatim from paper)

> Table 2: SVG refinement results on SVG-Sophia Code Refinement Benchmark. Metrics: DINO, SSIM, LPIPS, SR, Tokens. CTRL-S achieves the best performance across all evaluation metrics compared with state-of-the-art proprietary models, including GPT-5.2, Claude-Sonnet-4.5, and Gemini-3-Pro.

## Citation

```bibtex
@misc{wang2026ctrls,
  title={Reliable Reasoning in SVG-LLMs via Multi-Task Multi-Reward Reinforcement Learning},
  author={Haomin Wang et al. (2026)},
  year={2026},
  note={arXiv:2603.16189}
}
```

- arXiv: 2603.16189