Speech Likelihood Eval

---
name: speech-likelihood-eval
description: Evaluates the ability of generative latent variable models and autoregressive baselines to model speech audio distributions at varying temporal resolutions. It measures how well models capture intra-frame and inter-frame correlations in audio waveforms by optimizing likelihood objectives. Use when the user wants to benchmark on TIMIT, LibriSpeech, or asks about evaluating this task. Reports bits per frame (bpf).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2202.12707
  bibtex_key: [REDACTED]
  confidence: high
---

# speech-likelihood-eval

> Benchmarking Generative Latent Variable Models for Speech — Havtorn et al. (2022) (arXiv:2202.12707, 2022)

## What this evaluates

Evaluates the ability of generative latent variable models and autoregressive baselines to model speech audio distributions at varying temporal resolutions. It measures how well models capture intra-frame and inter-frame correlations in audio waveforms by optimizing likelihood objectives.

## Datasets

- **TIMIT** — total ?; splits: train (-1), val (-1), test (-1)
- **LibriSpeech** — total ?; splits: dev-clean (-1), dev-other (-1), test-clean (-1), test-other (-1)

## Metrics

- `bits per frame (bpf)` **(primary)** — range: other
  - Negative log-likelihood per audio frame converted to bits. For latent variable models, the one-sample Evidence Lower Bound (ELBO) is reported as an approximation. Lower values indicate better model fit and compression efficiency.

## Input / output format

**Input**: Stacked μ-law encoded PCM audio frames standardized to [-1, 1]. Each input x_t consists of s consecutive waveform frames x̃_{t:t+s}, where stack size s ∈ {1, 64, 256}. Audio is 16-bit depth at 16 kHz sample rate.

**Output**: Probability distribution (specifically a 10-component Discrete Mixture of Logistics, DMoL) over the discretized audio values for the stacked frames.

## Scoring recipe

```python
def compute_bpf(predictions, gold, stack_size, num_frames):
    # 1. Compute log-likelihood log p(x) using the model's output distribution (DMoL)
    # 2. For LVMs, use the one-sample ELBO as the log-likelihood estimate
    log_likelihood = compute_log_prob(predictions, gold)
    # 3. Convert to bits per frame: bpf = -log_p_x / (num_frames * log(2))
    bpf = -log_likelihood / (num_frames * np.log(2))
    # 4. Average bpf across all samples in the split
    return np.mean(bpf)
```

## Common pitfalls

- LVMs report the one-sample ELBO, which is a lower bound on the true log-likelihood; the paper marks these with ≤ in tables, meaning the true likelihood could be better (lower bpf).
- Stack size s drastically changes temporal resolution and likelihood values; models must be compared at identical s to avoid unfair comparisons.
- FLAC compression rates are reported on linear PCM, while models are trained on μ-law encoded audio, making direct compression-to-likelihood comparisons approximate.

## Evidence (verbatim from paper)

> We report likelihoods in units of bits per frame (bpf) as this yields a more interpretable and comparable likelihood than total likelihood in nats. It also has direct connections with information theory and compression (Shannon, 1948; Townsend et al., 2019). In units of bits per frame, lower is better. For LVMs, we report the one-sample ELBO.

## Citation

```bibtex
@misc{havtorn2022benchmarking,
  title={Benchmarking Generative Latent Variable Models for Speech},
  author={Havtorn et al. (2022)},
  year={2022},
  note={arXiv:2202.12707}
}
```

- arXiv: 2202.12707