Mnv 17 Eval

---
name: mnv-17-eval
description: Evaluates the ability of speech recognition models to jointly transcribe Mandarin speech and identify nonverbal vocalizations (NVs) like laughs or sighs. It also isolates the strict accuracy of NV event detection and measures whether adding NV recognition degrades core lexical transcription performance. Use when the user wants to benchmark on MNV-17, or asks about evaluating this task. Reports CER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.18196
  bibtex_key: mai2025mnv17
  confidence: high
---

# mnv-17-eval

> MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech — Mai et al. (2025) (arXiv:2509.18196, 2025)

## What this evaluates

Evaluates the ability of speech recognition models to jointly transcribe Mandarin speech and identify nonverbal vocalizations (NVs) like laughs or sighs. It also isolates the strict accuracy of NV event detection and measures whether adding NV recognition degrades core lexical transcription performance.

## Datasets

- **MNV-17** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `CER` **(primary)** — range: percent
  - Character Error Rate calculated after removing punctuation and spaces from transcripts. Each nonverbal vocalization (NV) label is treated as a single character. Averaged across all test samples.
- `NV Recognition Accuracy` — range: percent
  - Strict exact-match accuracy. A prediction is correct only if the type, count, and sequential order of all NV events exactly match the ground truth.

## Input / output format

**Input**: Mandarin speech audio recordings containing spoken text and nonverbal vocalizations.

**Output**: Text transcript containing lexical words and NV labels treated as single characters, plus explicit identification of NV event type, count, and sequential order.

## Scoring recipe

```python
def compute_cer(pred, gold):
    pred_clean = remove_punctuation_and_spaces(pred)
    return calculate_cer(pred_clean, gold)

def compute_nv_accuracy(pred, gold):
    pred_nvs = extract_nv_events(pred)
    gold_nvs = extract_nv_events(gold)
    correct = (pred_nvs.type == gold_nvs.type) and (pred_nvs.count == gold_nvs.count) and (pred_nvs.order == gold_nvs.order)
    return int(correct)
```

## Common pitfalls

- Treating NV labels as multiple characters or omitting them during CER calculation, which distorts error rates.
- Using partial matching for NV recognition (e.g., matching only type) instead of the required strict exact match on type, count, and order.
- Failing to strip NV tags from model predictions before calculating pure ASR CER in comparative analyses.

## Evidence (verbatim from paper)

> Following standard practice [[3], [40]], punctuation and spaces were removed from the transcripts before calculating the Character Error Rate (CER). Each NV label was treated as a single character, and the final CER was averaged across all samples in the test set.

## Citation

```bibtex
@misc{mai2025mnv17,
  title={MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech},
  author={Mai et al. (2025)},
  year={2025},
  note={arXiv:2509.18196}
}
```

- arXiv: 2509.18196