Meeting Asr Eval

---
name: meeting-asr-eval
description: This evaluation protocol assesses the robustness of automatic speech recognition (ASR) systems in multi-talker meeting scenarios under varying microphone configurations (close-talk, distant, and source-separated). It measures transcription accuracy across different levels of speaker overlap and acoustic conditions, while also analyzing how diarization errors propagate to downstream ASR performance. Use when the user wants to benchmark on LibriCSS, AMI, AliMeeting, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.05271
  bibtex_key: raj2022gss
  confidence: high
---

# meeting-asr-eval

> GPU-accelerated Guided Source Separation for Meeting Transcription — Desh Raj et al. (2022) (arXiv:2212.05271, 2022)

## What this evaluates

This evaluation protocol assesses the robustness of automatic speech recognition (ASR) systems in multi-talker meeting scenarios under varying microphone configurations (close-talk, distant, and source-separated). It measures transcription accuracy across different levels of speaker overlap and acoustic conditions, while also analyzing how diarization errors propagate to downstream ASR performance.

## Datasets

- **LibriCSS** — total ?; splits: dev (-1), test (-1); repo https://github.com/desh2608/gss
- **AMI** — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/desh2608/gss
- **AliMeeting** — total ?; splits: train (-1), eval (-1), test (-1); repo https://github.com/desh2608/gss

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate calculated as (Insertions + Deletions + Substitutions) / Total Words. Reported as a percentage.
- `CER` — range: percent
  - Character Error Rate calculated as (Insertions + Deletions + Substitutions) / Total Characters. Used specifically for the AliMeeting dataset.
- `DER` — range: percent
  - Diarization Error Rate measuring the proportion of time incorrectly assigned to speakers or background. Computed without collars for LibriCSS and AMI, and with a 0.25s collar for AliMeeting.
- `cpWER` — range: percent
  - Concatenated minimum-permutation Word Error Rate, aligning predicted and reference speaker turns before computing WER to isolate ASR performance from diarization errors.

## Input / output format

**Input**: Multi-channel audio recordings (individual headset, single distant, or GSS-enhanced multi-mic) organized into 10-minute mini-sessions or full meeting segments.

**Output**: Word-level or character-level transcriptions per audio segment, aligned with ground-truth speaker turns for cpWER/DER computation.

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_words = sum(len(ref) for ref in references)
    if total_words == 0: return 0.0
    errors = 0
    for pred, ref in zip(predictions, references):
        dist = levenshtein_distance(pred, ref)
        errors += dist
    return (errors / total_words) * 100
```

## Common pitfalls

- LibriCSS IHM is not real headset audio; it is simulated by concatenating LibriSpeech utterances, which may overestimate close-talk performance.
- AliMeeting uses CER instead of WER, and applies a 0.25s collar for DER computation, unlike LibriCSS and AMI which use no collar.
- GSS microphone setting uses different channel counts per dataset (7 channels for LibriCSS vs. 8 for AMI and AliMeeting), so direct cross-dataset comparison of channel counts is invalid.

## Evidence (verbatim from paper)

> For these experiments, we report diarization error rates (DER) and concatenated minimum-permutation WER (cpWER) [21] in order to analyze the impact of diarization errors on downstream ASR. We did not use any collars to compute DERs for LibriCSS and AMI, but a collar of 0.25 was used for AliMeeting following the original work.

## Citation

```bibtex
@misc{raj2022gss,
  title={GPU-accelerated Guided Source Separation for Meeting Transcription},
  author={Desh Raj et al. (2022)},
  year={2022},
  note={arXiv:2212.05271}
}
```

- arXiv: 2212.05271