Edacc Eval

---
name: edacc-eval
description: Evaluates automatic speech recognition (ASR) models on naturalistic, conversational English speech with diverse international accents. It probes the robustness of state-of-the-art ASR systems to real-world speaking conditions and accent variation compared to read-speech benchmarks like LibriSpeech. Use when the user wants to benchmark on EdAcc, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.18110
  bibtex_key: [REDACTED]
  confidence: high
---

# edacc-eval

> The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR — Sanabria et al. (2023) (arXiv:2303.18110, 2023)

## What this evaluates

Evaluates automatic speech recognition (ASR) models on naturalistic, conversational English speech with diverse international accents. It probes the robustness of state-of-the-art ASR systems to real-world speaking conditions and accent variation compared to read-speech benchmarks like LibriSpeech.

## Datasets

- **EdAcc** — total ?; splits: dev (-1), test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate computed at the conversation level. Standard formula: (Substitutions + Deletions + Insertions) / Total Reference Words.

## Input / output format

**Input**: Audio recordings of dyadic video call conversations, decoded in 30-second chunks.

**Output**: Transcribed text for each audio chunk/conversation.

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0
```

## Common pitfalls

- WER is aggregated at the conversation level rather than per utterance, which can skew averages if conversation lengths vary significantly.
- Accent labels are self-reported by speakers and not verified ground truth, making it difficult to establish a definitive 'true' label for evaluation.
- Models decode audio in fixed 30-second segments, which may introduce boundary artifacts not present in end-to-end streaming evaluation.

## Evidence (verbatim from paper)

> We start by measuring the general complexity of EdAcc by computing WER on development, and test sets at the conversation level. Because we can only compute one WER for each conversation, we limit this analysis to conversations where both speakers use the same English variety.

## Citation

```bibtex
@misc{sanabria2023edacc,
  title={The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR},
  author={Sanabria et al. (2023)},
  year={2023},
  note={arXiv:2303.18110}
}
```

- arXiv: 2303.18110