Sam Audio Eval

---
name: sam-audio-eval
description: Evaluates audio source separation capabilities conditioned on text, visual masks, or temporal spans. It probes open-vocabulary extraction, speaker/music/instrument isolation, and cross-modal grounding in both studio and in-the-wild settings. Use when the user wants to benchmark on SAM Audio Evaluation Set, MUSDB18, or asks about evaluating this task. Reports separation fidelity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.18099
  bibtex_key: shi2025samaudio
  confidence: high
---

# sam-audio-eval

> SAM Audio: Segment Anything in Audio — Shi et al. (2025) (arXiv:2512.18099, 2025)

## What this evaluates

Evaluates audio source separation capabilities conditioned on text, visual masks, or temporal spans. It probes open-vocabulary extraction, speaker/music/instrument isolation, and cross-modal grounding in both studio and in-the-wild settings.

## Datasets

- **SAM Audio Evaluation Set** — total ?; splits: test (-1)
- **MUSDB18** — total ?; splits: test (-1)

## Metrics

- `separation fidelity` **(primary)** — range: other
  - Not explicitly defined in the provided text; refers to the model's ability to isolate the target source without perceptual enhancement or reverb suppression.
- `subjective evaluation` — range: other
  - Human ratings on ~10s filtered clips to assess separation quality and prompt alignment.
- `CLAP similarity` — range: [0, 1]
  - Cosine similarity between audio and text embeddings using the CLAP model, used for speaker separation baseline comparison and inference re-ranking.
- `span IoU` — range: [0, 1]
  - Intersection over Union between predicted and ground-truth temporal spans for span-prompted tasks.

## Input / output format

**Input**: Audio mixture clip (~10s or up to 30s) paired with conditioning: a text prompt, a pair of video masks with the raw video, or temporal spans indicating target source activity.

**Output**: Separated audio track/stem corresponding to the target source specified by the prompt.

## Scoring recipe

```python
def evaluate(audio_mixture, prompt, ground_truth):
    pred_stem = model.predict(audio_mixture, prompt)
    fidelity = compute_separation_fidelity(pred_stem, ground_truth)
    subjective = human_rate(pred_stem)
    
    # Inference re-ranking scores (not final eval metrics)
    if prompt.type == 'text':
        rank = 1.0 * clap_score(pred_stem, prompt.text) + 5.0 * judge_score(pred_stem)
    elif prompt.type == 'span':
        rank = compute_iou(pred_stem.span, ground_truth.span)
    elif prompt.type == 'visual':
        rank = imagebind_similarity(pred_stem, prompt.video_mask)
    return fidelity, subjective, rank
```

## Common pitfalls

- Baselines are proprietary APIs; the paper explicitly disables their post-processing/enhancement modules to isolate separation fidelity, which may differ from standard API usage.
- Visual-prompted baselines (e.g., AV-Mossformer2) fail preprocessing on ~20% of evaluation videos, requiring sample exclusion that isn't always transparent.
- Evaluation clips are truncated to ~10s for subjective testing, which may not capture long-context separation performance.

## Evidence (verbatim from paper)

> To ensure fair comparison, we disable post-processing or enhancement modules in the baselines, as our evaluation focuses on separation fidelity—the model's ability to isolate speech content—rather than perceptual enhancement or reverb suppression.

## Citation

```bibtex
@misc{shi2025samaudio,
  title={SAM Audio: Segment Anything in Audio},
  author={Shi et al. (2025)},
  year={2025},
  note={arXiv:2512.18099}
}
```

- arXiv: 2512.18099