Semascore

---
name: semascore
description: Evaluates automatic speech recognition (ASR) transcription quality by measuring segment-wise semantic similarity and error weighting. It specifically probes robustness on disordered, noisy, and accented speech, testing alignment with human judgments and downstream NLU task metrics. Use when the user has predictions and gold and needs to compute SeMaScore.
metadata:
  skill_kind: metric
  source_arxiv: 2401.07506
  bibtex_key: zitha2024semascore
  confidence: high
---

# semascore

> SeMaScore : a new evaluation metric for automatic speech recognition tasks — Zitha Sasindran et al. (2024) (arXiv:2401.07506, 2024)

## What this evaluates

Evaluates automatic speech recognition (ASR) transcription quality by measuring segment-wise semantic similarity and error weighting. It specifically probes robustness on disordered, noisy, and accented speech, testing alignment with human judgments and downstream NLU task metrics.

## Datasets

- **Torgo** — total 600; splits: test (600)
- **Voicebank-DEMAND noisy speech testset** — total ?; splits: test (-1)
- **ATIS corpus (curated TTS)** — total 500; splits: test (500)

## Metrics

- `SeMaScore` **(primary)** — range: [0, 1]
  - Combines error-based metrics (e.g., MER) with semantic similarity computed via contextual embeddings (deberta-large-mnli), using Levenshtein-based sentence alignment to map hypothesis to reference segments.
- `BERTScore` — range: [0, 1]
  - Token-level cosine similarity between contextual embeddings of hypothesis and reference, aggregated across all token pairs.

## Input / output format

**Input**: Audio speech utterance (or pre-transcribed hypothesis text) and corresponding ground truth reference text.

**Output**: A scalar similarity score per utterance or segment, plus optional alignment mappings.

## Scoring recipe

```python
def compute_semascore(hypothesis, reference, embedder='deberta-large-mnli'):
    # 1. Align hypothesis and reference segments using Levenshtein/edit distance
    aligned_pairs = levenshtein_align(hypothesis, reference)
    # 2. Compute semantic similarity for each aligned segment pair
    sims = [cosine_similarity(embedder(seg_h), embedder(seg_r)) for seg_h, seg_r in aligned_pairs]
    # 3. Calculate error rate (e.g., MER)
    err_rate = compute_mer(hypothesis, reference)
    # 4. Combine similarity and error weighting
    score = weighted_combine(sims, err_rate)
    return score
```

## Common pitfalls

- BERTScore assigns high similarity scores to hypotheses that are semantically different from the ground truth in noisy or disordered speech.
- SeMaScore requires Levenshtein-based segment alignment rather than token-level mapping, which fundamentally changes score aggregation.
- Human assessment categories (0-2) are used only for correlation analysis, not as a direct metric output.

## Evidence (verbatim from paper)

> All contextual embeddings were obtained using pre-trained deberta-large-mnli model. Revisiting the examples in Table [1](#S1.T1 "Table 1 ‣ 1 Introduction and motivation ‣ SeMaScore : a new evaluation metric for automatic speech recognition tasks") where BERTscore fails, we can see that SeMaScore evaluated the hypotheses better by penalizing them appropriately.

## Citation

```bibtex
@misc{zitha2024semascore,
  title={SeMaScore : a new evaluation metric for automatic speech recognition tasks},
  author={Zitha Sasindran et al. (2024)},
  year={2024},
  note={arXiv:2401.07506}
}
```

- arXiv: 2401.07506