Mgb3 Eval

---
name: mgb3-eval
description: Evaluates automatic speech recognition and Arabic dialect identification in uncontrolled, real-world settings with high dialectal and genre diversity. It probes robustness to orthographic variability and low-resource conditions by measuring transcription accuracy against multiple human references. Use when the user wants to benchmark on MGB-3, or asks about evaluating this task. Reports MR-WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1709.07276
  bibtex_key: ali2017mgb3
  confidence: high
---

# mgb3-eval

> Speech Recognition Challenge in the Wild: Arabic MGB-3 — Ali et al. (2017) (arXiv:1709.07276, 2017)

## What this evaluates

Evaluates automatic speech recognition and Arabic dialect identification in uncontrolled, real-world settings with high dialectal and genre diversity. It probes robustness to orthographic variability and low-resource conditions by measuring transcription accuracy against multiple human references.

## Datasets

- **MGB-3** — total ?; splits: dev (1279)

## Metrics

- `MR-WER` **(primary)** — range: percent
  - Multi-Reference Word Error Rate. Computes standard WER against each available manual transcription and returns the minimum value. A recognized word is accepted if it appears in any reference at the same position.
- `WER` — range: percent
  - Standard Word Error Rate measuring the percentage of insertions, deletions, and substitutions relative to a single reference transcription.
- `Accuracy` — range: percent
  - Percentage of correctly predicted dialect labels in the Arabic Dialect Identification (ADI) task across five classes.

## Input / output format

**Input**: Raw audio speech signal (baseline systems may also use extracted lexical or i-vector features).

**Output**: For ASR: a transcribed text string. For ADI: a single dialect label from {Egyptian, Levantine, Gulf, North African, MSA}.

## Scoring recipe

```python
def compute_mr_wer(hypothesis, references):
    wers = []
    for ref in references:
        wers.append(compute_standard_wer(hypothesis, ref))
    return min(wers)

# For ADI task:
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Inter-annotator disagreement is high (~13%) due to the lack of standardized orthography in dialectal Arabic, making single-reference WER misleading.
- Systems trained on MGB-2 perform poorly on MGB-3 without adaptation to the changed genre and dialect characteristics.
- Surface text normalization must be applied before evaluation to account for orthographic variability across annotators.

## Evidence (verbatim from paper)

> For the MGB-3 Challenge, we investigated using MR-WER. This metric is based on comparing the recognized text against multiple manual transcriptions of the speech signal, which are all considered valid references. This approach thus accepts a recognized word if any of the references include it in the same form.

## Citation

```bibtex
@misc{ali2017mgb3,
  title={Speech Recognition Challenge in the Wild: Arabic MGB-3},
  author={Ali et al. (2017)},
  year={2017},
  note={arXiv:1709.07276}
}
```

- arXiv: 1709.07276