Dgfnet Av Sep Eval

---
name: dgfnet-av-sep-eval
description: Evaluates audio-visual models on their ability to separate target musical instrument sounds from mixed audio using synchronized video cues. It probes cross-modal feature alignment and dynamic fusion of audio and visual signals for source separation in complex environments. Use when the user wants to benchmark on MUSIC, MUSIC-21, or asks about evaluating this task. Reports SDR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.21366
  bibtex_key: yu2025dgfnet
  confidence: high
---

# dgfnet-av-sep-eval

> DGFNet: End-to-End Audio-Visual Source Separation Based on Dynamic Gating Fusion — Yu et al. (2025) (arXiv:2504.21366, 2025)

## What this evaluates

Evaluates audio-visual models on their ability to separate target musical instrument sounds from mixed audio using synchronized video cues. It probes cross-modal feature alignment and dynamic fusion of audio and visual signals for source separation in complex environments.

## Datasets

- **MUSIC** — total 685; splits: test (-1)
- **MUSIC-21** — total 1365; splits: test (-1)

## Metrics

- `SDR` **(primary)** — range: other
  - Signal-to-Distortion Ratio computed via mir_eval.separation.bss_eval_sources. Measures the ratio of target signal energy to total distortion energy (interference + artifacts + noise).
- `SIR` — range: other
  - Signal-to-Interference Ratio computed via mir_eval. Measures the ratio of target signal energy to interference energy from other sources.
- `SAR` — range: other
  - Signal-to-Artifacts Ratio computed via mir_eval. Measures the ratio of target signal energy to artifacts introduced by the separation algorithm.

## Input / output format

**Input**: Synchronized audio and video clips. Audio is downsampled to 11kHz (~6s clips), processed via STFT (1022-point Hann window, hop 256) and log-frequency resampled to 256xT spectrogram. Video is sampled at 1 FPS, 3 frames randomly selected, with detected objects resized/cropped to 224x224.

**Output**: Separated audio spectrogram or time-frequency mask for the target instrument(s).

## Scoring recipe

```python
import mir_eval
import numpy as np

def compute_metrics(y_true, y_pred):
    # y_true, y_pred: (n_channels, n_samples) numpy arrays
    sdr, sir, sar, perm = mir_eval.separation.bss_eval_sources(
        y_true, y_pred, compute_permutation=True
    )
    return {
        'SDR': float(np.mean(sdr)),
        'SIR': float(np.mean(sir)),
        'SAR': float(np.mean(sar))
    }
```

## Common pitfalls

- Missing video portions in the public MUSIC/MUSIC-21 datasets force reliance on reported results from Chen et al. (2023) for several baselines, which may limit direct reproducibility of the full comparison table.
- Audio preprocessing uses non-standard parameters for speech separation (11kHz downsampling, 1022-point STFT), so metrics are not directly comparable to standard audio benchmarks like WSJ0 or LibriSpeech.
- Visual feature extraction depends on external object detectors (Faster R-CNN or Detic); performance bottlenecks may stem from detection errors rather than the fusion module itself.

## Evidence (verbatim from paper)

> We used three common evaluation metrics: Signal-to-Distortion Ratio (SDR), Signal-to-Interference Ratio (SIR), and Signal-to-Artifacts Ratio (SAR), and evaluated the results using the mir_eval library (Raffel et al., [2014]). Higher values for all metrics indicate better performance. Regarding the SDR score, our method outperforms baseline model (Chen et al., [2023]) by 0.62 dB.

## Citation

```bibtex
@misc{yu2025dgfnet,
  title={DGFNet: End-to-End Audio-Visual Source Separation Based on Dynamic Gating Fusion},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2504.21366}
}
```

- arXiv: 2504.21366