Librispeech Wer Eval

---
name: librispeech-wer-eval
description: Evaluates the ability of a semantic-aware speech-to-text transmission system to accurately reconstruct text from speech signals under noisy communication channels (AWGN and Rayleigh). It probes semantic feature extraction, redundancy removal, and robustness to channel noise. Use when the user wants to benchmark on Librispeech, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2202.03211
  bibtex_key: han2022semantic
  confidence: high
---

# librispeech-wer-eval

> Semantic-aware Speech to Text Transmission with Redundancy Removal — Tianxiao Han et al. (2022) (arXiv:2202.03211, 2022)

## What this evaluates

Evaluates the ability of a semantic-aware speech-to-text transmission system to accurately reconstruct text from speech signals under noisy communication channels (AWGN and Rayleigh). It probes semantic feature extraction, redundancy removal, and robustness to channel noise.

## Datasets

- **Librispeech** — total ?; splits: train (-1), test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate computed by comparing the predicted transcription G~ against the ground truth transcription G. Standard substitution, deletion, and insertion errors are normalized by the reference length and expressed as a percentage.
- `sentence similarity` — range: [0, 1]
  - A metric measuring the semantic alignment between the predicted and ground truth sentences, computed alongside WER to assess content preservation beyond exact word matching.
- `transmission efficiency` — range: other
  - Average number of transmitted symbols per sentence, calculated from the output of the channel encoder after redundancy removal. Lower values indicate higher efficiency.

## Input / output format

**Input**: Raw speech signals (converted to spectrum sequences) and corresponding ground truth transcriptions. During inference, only speech signals are input to the semantic encoder.

**Output**: Reconstructed text transcription (G~) and associated metrics (WER, sentence similarity, symbol count).

## Scoring recipe

```python
def compute_metrics(predictions, ground_truths):
    wer_scores = []
    sim_scores = []
    for pred, gold in zip(predictions, ground_truths):
        wer_scores.append(calculate_wer(pred, gold))
        sim_scores.append(calculate_sentence_similarity(pred, gold))
    avg_wer = sum(wer_scores) / len(wer_scores)
    avg_sim = sum(sim_scores) / len(sim_scores)
    return avg_wer, avg_sim
```

## Common pitfalls

- Performance heavily depends on SNR regime; baselines degrade significantly at low SNR, making high-SNR comparisons insufficient.
- WER values may differ from prior DeepSC-SR reports (e.g., 20% vs 40%) due to specific feature extraction settings (40-dim fbank with derivatives).
- SE-DeepSC benchmark neglects semantic errors during transmission, leading to misleading upper-bound comparisons.

## Evidence (verbatim from paper)

> Algorithm 1 Testing algorithm of the proposed method. 1: Input: Speech signals and transcriptions G from dataset, fading channel h, noise w. 2: Set fading channel h = Rayleigh or AWGN 3: for each SNR value do 4: Generate Gaussian noise W under the SNR value. 5: Generate spectrum sequences S from input speech signals. 6: Output L from S by the semantic encoder. 7: Output X from L' by the channel encoder. 8: Transmit X and receive Y via (1). 9: Output L~ from Y by the channel decoder. 10: Output G~ from L~' by the semantic decoder. 11: end for 12: Output: compare G~ and G and compute WER scores and sentence similarity via 2

## Citation

```bibtex
@misc{han2022semantic,
  title={Semantic-aware Speech to Text Transmission with Redundancy Removal},
  author={Tianxiao Han et al. (2022)},
  year={2022},
  note={arXiv:2202.03211}
}
```

- arXiv: 2202.03211