Clapsep Eval

---
name: clapsep-eval
description: This benchmark evaluates query-conditioned target sound extraction (TSE), testing a model's ability to isolate a target audio source from a mixture using language captions or reference audio queries. It probes multi-modal query processing and positive/negative query valence across diverse acoustic environments and musical instruments. Use when the user wants to benchmark on AudioCaps, AudioSet, ESC-50, FSDKaggle2018, MUSIC21, or asks about evaluating this task. Reports SDRi, SISDRi.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.17455
  bibtex_key: ma2024clapsep
  confidence: high
---

# clapsep-eval

> CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extraction — Hao Ma et al. (2024) (arXiv:2402.17455, 2024)

## What this evaluates

This benchmark evaluates query-conditioned target sound extraction (TSE), testing a model's ability to isolate a target audio source from a mixture using language captions or reference audio queries. It probes multi-modal query processing and positive/negative query valence across diverse acoustic environments and musical instruments.

## Datasets

- **AudioCaps** — total ?; splits: test (4785)
- **AudioSet** — total ?; splits: eval (18869)
- **ESC-50** — total ?; splits: test (6500)
- **FSDKaggle2018** — total ?; splits: test (8000)
- **MUSIC21** — total ?; splits: test (19805)

## Metrics

- `SDRi` **(primary)** — range: other
  - SDRi = SDR(estimated_source, ground_truth) - SDR(mixture, ground_truth). Measures the improvement in signal-to-distortion ratio after separation.
- `SISDRi` **(primary)** — range: other
  - SISDRi = SISDR(estimated_source, ground_truth) - SISDR(mixture, ground_truth). Measures the improvement in scale-invariant signal-to-distortion ratio after separation.

## Input / output format

**Input**: A mixed audio waveform (target source + interference noise at 0 dB SNR) paired with a query, which can be either a text caption or a reference audio sample.

**Output**: A single extracted target sound source waveform corresponding to the query.

## Scoring recipe

```python
def score(predictions, gold, mixture):
    sdri = sdr(predictions, gold) - sdr(mixture, gold)
    sisdri = sisdr(predictions, gold) - sisdr(mixture, gold)
    return {'SDRi': sdri, 'SISDRi': sisdri}
```

## Common pitfalls

- Using evaluation mixtures with SNR levels other than the strictly specified 0 dB.
- Allowing query audio samples to be used in generating the evaluation mixtures, causing information leakage.
- Failing to resample all audio to 32kHz for consistent evaluation across datasets like ESC-50 and FSDKaggle2018.
- Confusing positive vs. negative query valence, which significantly impacts performance reporting.

## Evidence (verbatim from paper)

> Following previous works [[19], [20]], we use signal-to-distortion ratio improvement (SDRi) and scale-invariant signal-to-distortion ratio improvement (SISDRi) as the evaluation metrics. They indicate to what extent SDR and SISDR ... are improved by sound separation. They are defined as follows, SDRi(x̂, x̃, x) = SDR(x̂, x) - SDR(x̃, x), SISDRi(x̂, x̃, x) = SISDR(x̂, x) - SISDR(x̃, x), where x̂, x̃ and x denote the extracted sound source, sound mixture, and ground truth source, respectively.

## Citation

```bibtex
@misc{ma2024clapsep,
  title={CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extraction},
  author={Hao Ma et al. (2024)},
  year={2024},
  note={arXiv:2402.17455}
}
```

- arXiv: 2402.17455