Crowdspeech Eval

---
name: crowdspeech-eval
description: Evaluates algorithms for aggregating multiple noisy, crowdsourced transcriptions of the same audio recording into a single high-quality reference. It probes how well methods handle sequential textual noise, estimate worker reliability, and adapt across different audio quality domains. Use when the user wants to benchmark on CROWDSPEECH, VOXDIY, CROWDWSA2019, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2107.01091
  bibtex_key: pavlichenko2021crowdspeech
  confidence: high
---

# crowdspeech-eval

> CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription — Pavlichenko et al. (2021) (arXiv:2107.01091, 2021)

## What this evaluates

Evaluates algorithms for aggregating multiple noisy, crowdsourced transcriptions of the same audio recording into a single high-quality reference. It probes how well methods handle sequential textual noise, estimate worker reliability, and adapt across different audio quality domains.

## Datasets

- **CROWDSPEECH** — total ?; splits: dev-clean (-1), dev-other (-1), test-clean (-1), test-other (-1); repo https://github.com/Toloka/CrowdSpeech
- **VOXDIY** — total ?; splits: RU (-1)
- **CROWDWSA2019** — total ?; splits: J1 (-1), T1 (-1), T2 (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Average Word Error Rate computed as the normalized edit distance between the aggregated prediction and the ground truth reference transcription, averaged across all recordings in a split. Lower values indicate better performance.

## Input / output format

**Input**: A set of multiple crowdsourced text transcriptions (sequences) for a single audio recording.

**Output**: A single aggregated text transcription for that recording.

## Scoring recipe

```python
def compute_wer(predictions, references):
    wer_scores = []
    for pred, ref in zip(predictions, references):
        score = edit_distance(pred, ref) / max(len(ref.split()), 1)
        wer_scores.append(score)
    return sum(wer_scores) / len(wer_scores) * 100
```

## Common pitfalls

- Using WER on subjective tasks like machine translation (e.g., CROWDWSA2019) is inappropriate because there is no unique ground truth transcription.
- The Oracle baseline assumes access to ground truth to select the best worker annotation, which is unrealistic for deployment but serves only as an upper-bound estimate.
- Ignoring worker reliability/expertise when aggregating sequences can limit performance, as shown by the gap between simple voting methods and more advanced reliability-aware models.

## Evidence (verbatim from paper)

> Evaluation criterion is the average word error rate (WER) and lower values are better. To evaluate baseline methods, we run them on each of the datasets under consideration excluding the train set of CROWDSPEECH as its main purpose is model training. We then compute the mean value of WER (Word Error Rate) over all recordings in each dataset and report it in Table 4.

## Citation

```bibtex
@misc{pavlichenko2021crowdspeech,
  title={CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription},
  author={Pavlichenko et al. (2021)},
  year={2021},
  note={arXiv:2107.01091}
}
```

- arXiv: 2107.01091