Voxknesset Demographic Eval

---
name: voxknesset-demographic-eval
description: Evaluates whether the VoxKnesset dataset encodes meaningful demographic signals (gender, religion, birthplace) in pretrained speech representations. It probes the dataset's metadata quality and demographic coverage by training lightweight classifiers on extracted embeddings. Use when the user wants to benchmark on VoxKnesset Speaker-Attributed Longitudinal Subset, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.01270
  bibtex_key: marmor2026voxknesset
  confidence: high
---

# voxknesset-demographic-eval

> VoxKnesset: A Large-Scale Longitudinal Hebrew Speech Dataset for Aging Speaker Modeling — Marmor et al. (2026) (arXiv:2603.01270, 2026)

## What this evaluates

Evaluates whether the VoxKnesset dataset encodes meaningful demographic signals (gender, religion, birthplace) in pretrained speech representations. It probes the dataset's metadata quality and demographic coverage by training lightweight classifiers on extracted embeddings.

## Datasets

- **VoxKnesset Speaker-Attributed Longitudinal Subset** — total ?; splits: train (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Computed per demographic task to measure the Ridge classifier's discrimination ability between classes.

## Input / output format

**Input**: 1024-dimensional mean-pooled WavLM-Large embeddings extracted from audio segments ≥30 seconds.

**Output**: Predicted demographic category and continuous confidence scores for each class.

## Scoring recipe

```python
for task in ['gender', 'religion', 'birthplace']:
    clf = RidgeClassifier()
    clf.fit(X_train, y_train)
    y_prob = clf.predict_proba(X_test)[:, 1]
    auc_scores[task] = roc_auc_score(y_test, y_prob)
```

## Common pitfalls

- Splitting data by session instead of by speaker, which causes severe data leakage due to longitudinal overlap.
- Ignoring class imbalance in religion and birthplace categories, which can mislead accuracy-based evaluation but AUC remains robust.
- Using raw audio or different embedding models instead of the specified mean-pooled WavLM-Large features.

## Evidence (verbatim from paper)

> To validate that VoxKnesset captures meaningful speaker variation, we probe the demographic signal encoded in pretrained speech representations. We extract 1024-dimensional WavLM-Large embeddings (mean-pooled) and fit Ridge classifiers for gender, religion (4 classes), and birthplace (6 groups), see table 1.

| Task | #C | AUC | Maj. |
| --- | --- | --- | --- |
| Gender | 2 | 1.00 | .86 |
| Religion | 4 | .95 | .95 |
| Birthplace | 6 | .73 | .85 |

Table 1: Demographic prediction with WavLM-L embeddings (Ridge). #C = classes; Maj. = majority baseline.

## Citation

```bibtex
@misc{marmor2026voxknesset,
  title={VoxKnesset: A Large-Scale Longitudinal Hebrew Speech Dataset for Aging Speaker Modeling},
  author={Marmor et al. (2026)},
  year={2026},
  note={arXiv:2603.01270}
}
```

- arXiv: 2603.01270