Spqr Safety Eval

---
name: spqr-safety-eval
description: Evaluates the stability of safety alignment in text-to-image diffusion models after benign fine-tuning. It probes whether models suffer silent safety failures where utility remains high but safety degrades under distribution shifts like multilingual or domain-specific adaptation. Use when the user wants to benchmark on ViSU, I2P, RAB, or asks about evaluating this task. Reports R.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.19558
  bibtex_key: alam2025spqr
  confidence: high
---

# spqr-safety-eval

> SPQR: A Standardized Benchmark for Modern Safety Alignment Methods in Text-to-Image Diffusion Models — Mohammed Talha Alam et al. (2025) (arXiv:2511.19558, 2025)

## What this evaluates

Evaluates the stability of safety alignment in text-to-image diffusion models after benign fine-tuning. It probes whether models suffer silent safety failures where utility remains high but safety degrades under distribution shifts like multilingual or domain-specific adaptation.

## Datasets

- **ViSU** — total ?; splits: test (-1)
- **I2P** — total ?; splits: test (-1)
- **RAB** — total ?; splits: test (-1)

## Metrics

- `R` **(primary)** — range: [0, 1]
  - Robustness score computed by combining outputs from Nudenet and LLaVaGuard classifiers. Higher values indicate better preservation of safety alignment after fine-tuning.
- `SPQR` — range: [0, 1]
  - Composite leaderboard score aggregating four axes: Safety, Prompt Adherence, perceptual Quality, and Robustness (R). Used to compare trade-offs across methods.

## Input / output format

**Input**: Text prompts (harmful or benign) fed into a Stable Diffusion v1.5 model to generate images.

**Output**: Generated images, which are then scored by automated vision-language models (Nudenet, LLaVaGuard) and utility metrics.

## Scoring recipe

```python
def compute_R(images, prompts):
    nudenet_scores = run_nudenet(images)
    llava_scores = run_llavaguard(images, prompts)
    R = combine(nudenet_scores, llava_scores)  # Higher = more robust/safe
    return R

def compute_SPQR(safety_score, adherence_score, quality_score, R):
    SPQR = aggregate(safety_score, adherence_score, quality_score, R)
    return SPQR
```

## Common pitfalls

- Assuming stable prompt adherence or image quality implies preserved safety alignment.
- Evaluating robustness only on general English prompts without testing multilingual or domain-specific distribution shifts.
- Focusing solely on headline safety metrics, which can mask significant robustness degradation after fine-tuning.

## Evidence (verbatim from paper)

> We take the models that were fine-tuned on our Safe Benign Data and evaluate their final R↑ (Nudenet+LLaVaGuard) score on three different unseen harmful prompt datasets. High scores across all datasets show the failure is generalized.

## Citation

```bibtex
@misc{alam2025spqr,
  title={SPQR: A Standardized Benchmark for Modern Safety Alignment Methods in Text-to-Image Diffusion Models},
  author={Mohammed Talha Alam et al. (2025)},
  year={2025},
  note={arXiv:2511.19558}
}
```

- arXiv: 2511.19558