Librispeech Asr Correction Eval

---
name: librispeech-asr-correction-eval
description: Evaluates confidence-based filtering strategies for applying LLMs to post-hoc correction of ASR transcripts, measuring how well the system reduces transcription errors in low-confidence segments while preserving accurate outputs. Use when the user wants to benchmark on LibriSpeech, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.21414
  bibtex_key: naderi2024asrllm
  confidence: high
---

# librispeech-asr-correction-eval

> Towards interfacing large language models with ASR systems using confidence measures and prompting — Naderi et al. (2024) (arXiv:2407.21414, 2024)

## What this evaluates

Evaluates confidence-based filtering strategies for applying LLMs to post-hoc correction of ASR transcripts, measuring how well the system reduces transcription errors in low-confidence segments while preserving accurate outputs.

## Datasets

- **LibriSpeech** — total ?; splits: dev-clean (-1), dev-other (-1), test-clean (-1), test-other (-1); HF `librispeech_asr`

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate calculated as the minimum number of insertions, deletions, and substitutions required to transform the predicted transcript into the reference transcript, divided by the number of words in the reference transcript.

## Input / output format

**Input**: ASR transcript with associated sentence- and word-level confidence scores, processed through a confidence threshold filter before being passed to an LLM prompt for correction.

**Output**: Corrected ASR transcript (or the original unmodified transcript if confidence scores exceed the filtering threshold).

## Scoring recipe

```python
def calculate_wer(predictions, references):
    total_errors = 0
    total_ref_words = 0
    for pred, ref in zip(predictions, references):
        # Standard Levenshtein-based edit distance
        edit_dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += edit_dist
        total_ref_words += len(ref.split())
    return (total_errors / total_ref_words) * 100 if total_ref_words > 0 else 0.0
```

## Common pitfalls

- LLMs may introduce new errors into high-confidence transcripts if filtering thresholds are set too loosely.
- Whisper's confidence scores are averaged token log-probabilities, which may not perfectly correlate with actual word-level errors.
- Potential data contamination risk as the proprietary ChatGPT models may have been trained on LibriSpeech data.

## Evidence (verbatim from paper)

> We evaluate our proposed approach on the English LibriSpeech corpus [27] of audiobook recordings. We use the dev-clean and dev-other subsets for initial experiments and hyperparameter tuning and then report final results on the test-clean and test-other evaluation sets. Each of these subsets contains around 2500-3000 utterances. Speakers in the other portions are more challenging to recognize and lead to higher WERs.

## Citation

```bibtex
@misc{naderi2024asrllm,
  title={Towards interfacing large language models with ASR systems using confidence measures and prompting},
  author={Naderi et al. (2024)},
  year={2024},
  note={arXiv:2407.21414}
}
```

- arXiv: 2407.21414