Jam Alt Eval

---
name: jam-alt-eval
description: Evaluates automatic lyrics transcription (ALT) systems on readability-aware formatting, including punctuation, capitalization, line breaks, and background vocal annotations. It distinguishes errors by token type to measure how well models adhere to industry-standard musical semantics and prosodic structure. Use when the user wants to benchmark on Jam-ALT, Schubert Winterreise Dataset (SWD), or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.06370
  bibtex_key: cifka2024lyricstranscription
  confidence: high
---

# jam-alt-eval

> Lyrics Transcription for Humans: A Readability-Aware Benchmark — Cífka et al. (2024) (arXiv:2408.06370, 2024)

## What this evaluates

Evaluates automatic lyrics transcription (ALT) systems on readability-aware formatting, including punctuation, capitalization, line breaks, and background vocal annotations. It distinguishes errors by token type to measure how well models adhere to industry-standard musical semantics and prosodic structure.

## Datasets

- **Jam-ALT** — total ?; splits: test (-1)
- **Schubert Winterreise Dataset (SWD)** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: (substitutions + insertions + deletions) / reference length. Normalized by reference length.
- `WER'` — range: percent
  - Case-sensitive Word Error Rate: WER + case errors (hits with differing letter case counted as errors instead of hits).
- `FP` — range: percent
  - F-measure for punctuation tokens.
- `FB` — range: percent
  - F-measure for parentheses tokens.
- `FL` — range: percent
  - F-measure for line break tokens.
- `FS` — range: percent
  - F-measure for section break tokens.

## Input / output format

**Input**: Raw audio track (optionally pre-processed with HTDemucs for vocal isolation) and song language identifier.

**Output**: Textual lyrics with enforced line breaks, capitalization, punctuation, and annotations for parentheses/section breaks, following readability-aware annotation guidelines.

## Scoring recipe

```python
def compute_metrics(pred, ref):
    # Tokenize into words and special formatting tokens (P, B, L, S)
    ops = edit_distance(pred, ref)
    wer = (ops.sub + ops.ins + ops.del) / len(ref)
    case_errors = count_case_mismatches(pred, ref)
    wer_prime = wer + case_errors / len(ref)
    fp = f1_score(pred.P, ref.P)
    fb = f1_score(pred.B, ref.B)
    fl = f1_score(pred.L, ref.L)
    fs = f1_score(pred.S, ref.S)
    return {'WER': wer, "WER'": wer_prime, 'FP': fp, 'FB': fb, 'FL': fl, 'FS': fs}
```

## Common pitfalls

- Whisper models use stochastic decoding, so results must be averaged over multiple runs (e.g., 5) to be comparable.
- Providing the known language to Whisper often degrades performance compared to its auto-detection, contrary to typical ASR expectations.
- Using vocal separation (HTDemucs) as input can substantially degrade transcription accuracy for some models.
- The SWD subset contains obsolete German spelling that violates modern annotation guidelines, artificially inflating WER.

## Evidence (verbatim from paper)

> Table 1: Benchmark results (all metrics shown as percentages). WER is word error rate, $	ext{WER}'$ is case-sensitive WER, the rest are F-measures.

## Citation

```bibtex
@misc{cifka2024lyricstranscription,
  title={Lyrics Transcription for Humans: A Readability-Aware Benchmark},
  author={Cífka et al. (2024)},
  year={2024},
  note={arXiv:2408.06370}
}
```

- arXiv: 2408.06370