Naijas2st Eval

---
name: naijas2st-eval
description: Evaluates speech-to-text and speech-to-speech translation capabilities across low-resource Nigerian languages (Hausa, Igbo, Yorùbá, Nigerian Pidgin) and English. It specifically probes how well cascaded, end-to-end, and AudioLLM architectures handle multi-accent variations and bidirectional translation directions. Use when the user wants to benchmark on NaijaS2ST, or asks about evaluating this task. Reports SSA-COMET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.16287
  bibtex_key: maltais2026naijas2st
  confidence: high
---

# naijas2st-eval

> NaijaS2ST: A Multi-Accent Benchmark for Speech-to-Speech Translation in Low-Resource Nigerian Languages — Maltais et al. (2026) (arXiv:2604.16287, 2026)

## What this evaluates

Evaluates speech-to-text and speech-to-speech translation capabilities across low-resource Nigerian languages (Hausa, Igbo, Yorùbá, Nigerian Pidgin) and English. It specifically probes how well cascaded, end-to-end, and AudioLLM architectures handle multi-accent variations and bidirectional translation directions.

## Datasets

- **NaijaS2ST** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `SSA-COMET` **(primary)** — range: [0, 100]
  - Sentence-level semantic similarity metric based on COMET embeddings, adapted for speech translation. Scores higher for better semantic alignment between source and target, typically reported on a 0-100 scale.
- `ChrF` — range: [0, 100]
  - Character n-gram F-score measuring surface-level character overlap between reference and hypothesis. Highly sensitive to diacritics and spelling variations.
- `SpBLEU` — range: [0, 100]
  - Byte-level n-gram F-score for subword tokenization, used as an additional surface-form metric for translation quality.

## Input / output format

**Input**: Source speech audio in Hausa, Igbo, Yorùbá, Nigerian Pidgin, or English, recorded with either Naija or British RP accent.

**Output**: Target text transcript (for S2TT) or target speech audio (for S2ST), depending on the pipeline configuration.

## Scoring recipe

```python
if task == 'S2TT':
    ssa_comet = compute_ssa_comet(hypothesis_text, reference_text)
    chrf = compute_chrf(hypothesis_text, reference_text)
elif task == 'S2ST':
    transcribed_hypothesis = asr_model.generate_transcript(hypothesis_speech)
    ssa_comet = compute_ssa_comet(transcribed_hypothesis, reference_text)
    chrf = compute_chrf(transcribed_hypothesis, reference_text)
# Note: For S2ST, the ASR model (Omnilingual-ASR 1B or Naija-Omni) must first transcribe the generated speech before metric computation.
```

## Common pitfalls

- SSA-COMET can yield high semantic scores even when the model outputs the source language instead of the target language, due to its tolerance for lexical and surface-form mismatches.
- ChrF heavily penalizes languages with extensive diacritics (e.g., Yorùbá), making it an unreliable standalone metric for these languages and potentially masking genuine translation progress.
- Evaluation ASR models exhibit accent bias; using a base model like Omnilingual-ASR on Naija-accented speech artificially deflates S2ST scores unless the ASR is fine-tuned on the target accent (Naija-Omni).

## Evidence (verbatim from paper)

> For evaluation, we use Omnilingual-ASR 1B model for ASR on the speech outputs, then extract the SSA-COMET and ChrF scores from those transcripts. This discrepancy highlights a key limitation of embedding-based metrics: SSA-COMET is more tolerant to lexical and surface-form mismatches, whereas ChrF penalizes such deviations more directly.

## Citation

```bibtex
@misc{maltais2026naijas2st,
  title={NaijaS2ST: A Multi-Accent Benchmark for Speech-to-Speech Translation in Low-Resource Nigerian Languages},
  author={Maltais et al. (2026)},
  year={2026},
  note={arXiv:2604.16287}
}
```

- arXiv: 2604.16287