Spatial Audio Motion Eval

---
name: spatial-audio-motion-eval
description: Probes a model's ability to detect overlapping audio events in dynamic spatial recordings, estimate their direction of arrival and distance, and perform reasoning about moving sound sources. Use when the user wants to benchmark on STARS23, FOA-MEIR Derived, or asks about evaluating this task. Reports F-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.14666
  bibtex_key: [REDACTED]
  confidence: medium
---

# spatial-audio-motion-eval

> Spatial Audio Motion Understanding and Reasoning — Sridhar et al. (2025) (arXiv:2509.14666, 2025)

## What this evaluates

Probes a model's ability to detect overlapping audio events in dynamic spatial recordings, estimate their direction of arrival and distance, and perform reasoning about moving sound sources.

## Datasets

- **STARS23** — total 30000; splits: train (30000), test (-1)
- **FOA-MEIR Derived** — total 3000; splits: test (3000)

## Metrics

- `F-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for frame-wise audio event detection, adopted from the DCASE 2025 Task 3 baseline protocol.
- `DoA error` — range: degrees
  - Mean absolute angular difference (in degrees) between predicted and ground-truth direction of arrival.
- `Relative Distance Error` — range: [0, 1]
  - Mean absolute difference between predicted and ground-truth source distance divided by ground-truth distance.

## Input / output format

**Input**: 5-second stereo spatial audio clips containing up to three overlapping events, paired with text prompts for question answering.

**Output**: Frame-wise binary scores for audio tags, estimated DoA angles, estimated source distances, and textual answers (MCQ or Boolean) to spatial reasoning questions.

## Scoring recipe

```python
def evaluate(preds, gold):
    # preds, gold: dicts with keys 'tags', 'doa', 'dist'
    f1 = f1_score(gold['tags'], preds['tags'], average='macro')
    doa_err = np.mean(np.abs(gold['doa'] - preds['doa']))
    rde = np.mean(np.abs(gold['dist'] - preds['dist']) / gold['dist'])
    return {'F-score': f1, 'DoA error': doa_err, 'Relative Distance Error': rde}
```

## Common pitfalls

- Baseline models were fine-tuned on additional synthetic data, but this protocol strictly uses only the public STARS23 training set for reproducibility.
- FOA recordings must be converted to stereo format before evaluation to align with the DCASE 2025 Task 3 baseline protocol.
- Generalization is evaluated on a custom dataset of 9 unseen events spatialized via Spatial Scaper, not on the original FOA-MEIR test split.

## Evidence (verbatim from paper)

> Evaluation is performed on the STARSS23 test set. To assess generalization, we built our 9 unseen isolated sound events dataset using dry source recordings from FOA-MEIR dataset[5]. We spatialize the audioclips using Spatial Scaper[13], generating 3k five-second stereo clips with up to three overlapping events per frame. FOA outputs are converted to stereo following[1]. We adopt the multi-ACCDDOA loss, evaluation metrics-F-score, DoA error, and Relative Distance Error, as well as feature extraction hyperparameters from[1].

## Citation

```bibtex
@misc{sridhar2025spatial,
  title={Spatial Audio Motion Understanding and Reasoning},
  author={Sridhar et al. (2025)},
  year={2025},
  note={arXiv:2509.14666}
}
```

- arXiv: 2509.14666