Librispeech Pc Eval

---
name: librispeech-pc-eval
description: Evaluates the ability of end-to-end automatic speech recognition (ASR) models to correctly predict punctuation marks and word capitalization in transcribed speech. It specifically isolates punctuation-specific errors to enable fine-grained comparison between cascade and end-to-end architectures. Use when the user wants to benchmark on LibriSpeech-PC, or asks about evaluating this task. Reports Punctuation Error Rate (PER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.02943
  bibtex_key: meister2023librispeechpc
  confidence: high
---

# librispeech-pc-eval

> LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models — Meister et al. (2023) (arXiv:2310.02943, 2023)

## What this evaluates

Evaluates the ability of end-to-end automatic speech recognition (ASR) models to correctly predict punctuation marks and word capitalization in transcribed speech. It specifically isolates punctuation-specific errors to enable fine-grained comparison between cascade and end-to-end architectures.

## Datasets

- **LibriSpeech-PC** — total ?; splits: test (-1)

## Metrics

- `Punctuation Error Rate (PER)` **(primary)** — range: percent
  - Calculated as the normalized Levenshtein distance between the predicted and ground-truth sequences of punctuation tokens (periods, commas, question marks), isolating punctuation-specific errors.

## Input / output format

**Input**: Audio recordings paired with reference transcripts preprocessed to retain only periods, commas, and question marks, with words and punctuation separated by single spaces.

**Output**: Predicted text sequence with words and punctuation/capitalization tokens separated by spaces.

## Scoring recipe

```python
def compute_per(pred_text, gold_text):
    # Extract only punctuation tokens as per benchmark preprocessing
    pred_punct = [w for w in pred_text.split() if w in {'.', ',', '?'}]
    gold_punct = [w for w in gold_text.split() if w in {'.', ',', '?'}]
    # Compute Levenshtein distance
    dist = levenshtein_distance(pred_punct, gold_punct)
    # Normalize by gold length to get error rate
    return dist / len(gold_punct) if len(gold_punct) > 0 else 0.0
```

## Common pitfalls

- The benchmark restricts evaluation to only periods, commas, and question marks, ignoring other punctuation marks present in standard ASR datasets.
- Models must be evaluated on space-separated token sequences (e.g., 'done .') rather than raw concatenated text to match the preprocessing protocol.
- Comparing cascade and E2E models requires identical decoding settings (e.g., beam search with n-gram LMs) to avoid confounding factors.

## Evidence (verbatim from paper)

> It proposes a novel Punctuation Error Rate (PER) metric based on Levenshtein distance that isolates punctuation-specific errors, enabling fine-grained comparison between cascaded and end-to-end models.

## Citation

```bibtex
@misc{meister2023librispeechpc,
  title={LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models},
  author={Meister et al. (2023)},
  year={2023},
  note={arXiv:2310.02943}
}
```

- arXiv: 2310.02943