Neural Beamforming Adaptation Eval

---
name: neural-beamforming-adaptation-eval
description: Evaluates the robustness of a run-time adapted neural beamforming system for joint speech dereverberation and denoising under varying acoustic conditions, including different numbers of speakers, reverberation times, and SNRs. Use when the user wants to benchmark on Simulated Librispeech+DEMAND, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.22805
  bibtex_key: fujita2024runtime
  confidence: high
---

# neural-beamforming-adaptation-eval

> Run-Time Adaptation of Neural Beamforming for Robust Speech Dereverberation and Denoising — Fujita et al. (2024) (arXiv:2410.22805, 2024)

## What this evaluates

Evaluates the robustness of a run-time adapted neural beamforming system for joint speech dereverberation and denoising under varying acoustic conditions, including different numbers of speakers, reverberation times, and SNRs.

## Datasets

- **Simulated Librispeech+DEMAND** — total ?; splits: train (36000), val (3600), test (210)

## Metrics

- `WER` **(primary)** — range: [0, 1] or percent
  - Word Error Rate: edit distance between predicted and reference transcriptions divided by reference length.
- `SDR` — range: dB
  - Signal-to-Distortion Ratio: ratio of target signal energy to distortion energy (noise, artifacts, interference) in decibels.
- `STOI` — range: [0, 1]
  - Short-Time Objective Intelligibility: temporal correlation-based metric measuring speech envelope similarity between clean and enhanced signals.
- `PESQ` — range: [-0.5, 4.5]
  - Perceptual Evaluation of Speech Quality: ITU-T P.862 standard mapping to a MOS-like score based on perceptual speech quality.
- `SRMR` — range: dB
  - Speech-to-Reverberation Modulation Energy Ratio: ratio of modulation energy in speech-dominant bands to reverberation-dominant bands.

## Input / output format

**Input**: 7-channel STFT coefficients (window=1024, hop=256) of the mixture signal, concatenated with log magnitude, inter-channel phase differences, and delay-sum beamforming output. Target DOA is provided as auxiliary input.

**Output**: Time-frequency (TF) mask for weighted power minimization distortionless response (WPD) beamforming, plus adapted filter parameters for WPE and FastMNMF.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    wer = compute_wer(predictions.transcription, gold.transcription)
    sdr = compute_sdr(predictions.waveform, gold.waveform)
    stoi = compute_stoi(predictions.waveform, gold.waveform)
    pesq = compute_pesq(predictions.waveform, gold.waveform)
    srmer = compute_srmer(predictions.waveform, gold.waveform)
    return {'WER': wer, 'SDR': sdr, 'STOI': stoi, 'PESQ': pesq, 'SRMR': srmer}
```

## Common pitfalls

- Evaluation uses only the second half (~4 minutes) of each 8-minute test recording; the first half is strictly reserved for fine-tuning the mask estimator.
- No ground-truth speech is available during run-time adaptation; the system relies on pseudo-ground-truth estimates from FastMNMF.
- Test noise is strictly from DEMAND's 'PSTATION' environment, while training/validation noise explicitly excludes it to prevent data leakage.

## Evidence (verbatim from paper)

> The evaluation compared different amounts of fine-tuning data (30, 60, 120, and 240 seconds) against several key metrics. These metrics include word error rate (WER), signal-to-distortion ratio (SDR), short-time objective intelligibility (STOI), perceptual evaluation of speech quality (PESQ), and speech-to-reverberation modulation energy ratio (SRMR).

## Citation

```bibtex
@misc{fujita2024runtime,
  title={Run-Time Adaptation of Neural Beamforming for Robust Speech Dereverberation and Denoising},
  author={Fujita et al. (2024)},
  year={2024},
  note={arXiv:2410.22805}
}
```

- arXiv: 2410.22805