Singverse Eval

---
name: singverse-eval
description: Evaluates singing voice enhancement models on real-world acoustic scenarios, measuring their ability to improve perceptual quality and content intelligibility of degraded singing vocals without degrading speech capabilities. Use when the user wants to benchmark on SingVERSE, or asks about evaluating this task. Reports perceptual quality.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.20969
  bibtex_key: jiang2025singverse
  confidence: high
---

# singverse-eval

> SingVERSE: A Diverse, Real-World Benchmark for Singing Voice Enhancement — Jiang et al. (2025) (arXiv:2509.20969, 2025)

## What this evaluates

Evaluates singing voice enhancement models on real-world acoustic scenarios, measuring their ability to improve perceptual quality and content intelligibility of degraded singing vocals without degrading speech capabilities.

## Datasets

- **SingVERSE** — total 3971; splits: test (3971)

## Metrics

- `perceptual quality` **(primary)** — range: not specified
  - Not explicitly defined in the provided section; measures the fidelity and naturalness of the enhanced singing voice relative to the clean reference.
- `content intelligibility` — range: not specified
  - Not explicitly defined in the provided section; measures the clarity and understandability of the enhanced singing lyrics relative to the clean reference.

## Input / output format

**Input**: Degraded real-world singing audio clips from 19 diverse acoustic environments, paired with studio-quality clean references.

**Output**: Enhanced singing audio clips.

## Scoring recipe

```python
def evaluate(predictions, gold):
    # predictions: enhanced audio, gold: clean reference
    perceptual_score = compute_perceptual_quality(predictions, gold)
    intelligibility_score = compute_intelligibility(predictions, gold)
    return perceptual_score, intelligibility_score
```

## Common pitfalls

- Models trained on synthetic data often fail to generalize to the diverse real-world acoustic scenarios covered in the benchmark.
- Optimizing solely for perceptual quality can degrade content intelligibility, highlighting a critical trade-off that must be balanced during evaluation.

## Evidence (verbatim from paper)

> A controlled experiment reveals a statistically significant domain mismatch when speech enhancement models are applied to singing, with performance drops in perceptual quality and intelligibility across all models.

## Citation

```bibtex
@misc{jiang2025singverse,
  title={SingVERSE: A Diverse, Real-World Benchmark for Singing Voice Enhancement},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2509.20969}
}
```

- arXiv: 2509.20969