Imasc Eval

---
name: imasc-eval
description: Evaluates the perceptual quality and naturalness of synthesized Malayalam speech generated by a multi-speaker TTS model trained on the IMaSC corpus. It probes the model's ability to capture agglutinative morphology, phonemic orthography, and diverse prosodic styles through subjective human listening tests. Use when the user wants to benchmark on IMaSC, or asks about evaluating this task. Reports Mean Opinion Score (MOS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.12796
  bibtex_key: gopinath2022imasc
  confidence: high
---

# imasc-eval

> IMaSC -- ICFOSS Malayalam Speech Corpus — Gopinath et al. (2022) (arXiv:2211.12796, 2022)

## What this evaluates

Evaluates the perceptual quality and naturalness of synthesized Malayalam speech generated by a multi-speaker TTS model trained on the IMaSC corpus. It probes the model's ability to capture agglutinative morphology, phonemic orthography, and diverse prosodic styles through subjective human listening tests.

## Datasets

- **IMaSC** — total 34473; splits: test (-1)

## Metrics

- `Mean Opinion Score (MOS)` **(primary)** — range: [1, 5]
  - Average rating on a 5-point Likert scale for naturalness (5=excellent, 4=good, 3=fair, 2=poor, 1=bad) assigned by human listeners. Computed as the mean score across all participants and all evaluated audio samples.

## Input / output format

**Input**: Malayalam text sentences (tokenized as characters) for synthesis; corresponding ground-truth audio samples for reference.

**Output**: Synthesized audio waveforms generated by the VITS TTS model.

## Scoring recipe

```python
def compute_mos(predictions, gold, num_participants=20):
    total_score = 0
    total_ratings = 0
    for sample in predictions:
        ratings = get_human_ratings(sample, num_participants)
        total_score += sum(ratings)
        total_ratings += num_participants
    return total_score / total_ratings
```

## Common pitfalls

- The evaluation uses a very small test set (only 10 synthesized + 2 ground truth sentences per speaker), which may not capture full corpus diversity or speaker variability.
- MOS is highly subjective and sensitive to participant demographics, listening environment, and prior exposure to Malayalam, none of which are controlled or reported.
- The paper averages MOS across speakers and participants without reporting per-speaker variance or statistical significance testing.

## Evidence (verbatim from paper)

> We conducted a crowd-sourced Mean Opinion Score (MOS) test with 20 participants for evaluating the models. 10 sentences were randomly selected from the test dataset to be synthesized, and 2 sentences, along with their corresponding audio, were chosen for evaluating ground truth. In the survey, each speaker thus had 12 text-audio pairs to be evaluated, repeated for all 8 speakers for a total of 96 questions. The different audio samples were each scored on a 5-point scale for naturalness, with 5 being excellent, 4 being good, 3 being fair, 2 being poor, and 1 being bad.

## Citation

```bibtex
@misc{gopinath2022imasc,
  title={IMaSC -- ICFOSS Malayalam Speech Corpus},
  author={Gopinath et al. (2022)},
  year={2022},
  note={arXiv:2211.12796}
}
```

- arXiv: 2211.12796