Nim4 Asr Eval

---
name: nim4-asr-eval
description: Evaluates automatic speech recognition performance across diverse acoustic and linguistic domains, including English, Mandarin, dialects, code-switching, and in-car conversational scenarios. It measures transcription accuracy and hallucination rates to assess model robustness, latency, and customization capabilities. Use when the user wants to benchmark on LibriSpeech, VoxPopuli, MLS-English, AISHELL-1, AISHELL-2, AISHELL-2021-Eval, WeNetSpeech, SpeechIO, WeNetSpeech-Chuan, WeNetSpeech-Yue, KeSpeech, CS-Dialogue, ASCEND, M4Singer, Internal POI Benchmarks, Internal Media Benchmarks, Internal Device Control, Internal Conversational, or asks about evaluating this task. Reports WER, CER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18105
  bibtex_key: xie2026nim4asr
  confidence: high
---

# nim4-asr-eval

> NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR — Yuan Xie et al. (2026) (arXiv:2604.18105, 2026)

## What this evaluates

Evaluates automatic speech recognition performance across diverse acoustic and linguistic domains, including English, Mandarin, dialects, code-switching, and in-car conversational scenarios. It measures transcription accuracy and hallucination rates to assess model robustness, latency, and customization capabilities.

## Datasets

- **LibriSpeech** — total ?; splits: dev (-1), test (-1)
- **VoxPopuli** — total ?; splits: dev (-1), test (-1)
- **MLS-English** — total ?; splits: test (-1)
- **AISHELL-1** — total ?; splits: dev (-1), test (-1)
- **AISHELL-2** — total ?; splits: ios (-1), test (-1)
- **AISHELL-2021-Eval** — total ?; splits: A (-1), C (-1), D (-1)
- **WeNetSpeech** — total ?; splits: meeting (-1), net (-1)
- **SpeechIO** — total ?; splits: test (-1)
- **WeNetSpeech-Chuan** — total ?; splits: easy (-1), hard (-1)
- **WeNetSpeech-Yue** — total ?; splits: short (-1), long (-1)
- **KeSpeech** — total ?; splits: test (-1)
- **CS-Dialogue** — total ?; splits: test (-1)
- **ASCEND** — total ?; splits: test (-1)
- **M4Singer** — total ?; splits: test (-1)
- **Internal POI Benchmarks** — total ?; splits: City A (-1), City B (-1), City C (-1), City D (-1)
- **Internal Media Benchmarks** — total ?; splits: Music (-1), Video (-1), Radio (-1)
- **Internal Device Control** — total ?; splits: Vehicle control (-1)
- **Internal Conversational** — total ?; splits: Vehicle-domain chat (-1), Multi-domain chat (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: ratio of substitutions, deletions, and insertions to the total number of words in the reference transcription. Used for English benchmarks.
- `CER` **(primary)** — range: percent
  - Character Error Rate: ratio of substitutions, deletions, and insertions to the total number of characters in the reference transcription. Used for Mandarin, dialects, lyrics, and code-switched benchmarks.
- `Hallucination Rate` — range: percent
  - Ratio of hallucinated samples to total samples. A sample is hallucinated if its transcription length exceeds the ground-truth length by over 50% with negligible lexical overlap.
- `POI Recall` — range: percent
  - Proportion of Point of Interest (POI) entities correctly recognized in the transcription output.

## Input / output format

**Input**: Audio recordings paired with ground-truth transcriptions.

**Output**: Text transcriptions generated by the ASR model.

## Scoring recipe

```python
def compute_wer_cer(pred, ref):
    pred_norm = normalize_text(pred)
    ref_norm = normalize_text(ref)
    ops = levenshtein_operations(pred_norm, ref_norm)
    errors = ops['substitutions'] + ops['deletions'] + ops['insertions']
    total = len(ref_norm.split()) if is_english else len(ref_norm)
    return (errors / total) * 100

def compute_hallucination_rate(predictions, references):
    hallucinated_count = 0
    for pred, ref in zip(predictions, references):
        if len(pred) > len(ref) * 1.5 and lexical_overlap(pred, ref) < threshold:
            hallucinated_count += 1
    return (hallucinated_count / len(predictions)) * 100
```

## Common pitfalls

- Text normalization via WeTextProcessing is mandatory for both predictions and references; skipping it inflates error rates and breaks fair comparison.
- Baselines are strictly evaluated in offline mode, while NIM4-ASR reports both offline and streaming results; comparing streaming NIM4-ASR against offline baselines is invalid.
- Hallucination rate evaluation explicitly excludes WeNetSpeech Meeting, SpeechIO, MLS-English, and WeNetSpeech Net due to zero hallucinations or unreliable annotations.

## Evidence (verbatim from paper)

> We report Word Error Rate (WER) for English benchmarks, and Character Error Rate (CER) for Mandarin, Chinese dialect, lyrics, and code-switched Chinese-English benchmarks. ... Specifically, a sample is classified as hallucinated if its transcription exceeds the ground-truth length by over 50% with negligible lexical overlap.

## Citation

```bibtex
@misc{xie2026nim4asr,
  title={NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR},
  author={Yuan Xie et al. (2026)},
  year={2026},
  note={arXiv:2604.18105}
}
```

- arXiv: 2604.18105