Speakersleuth Eval

---
name: speakersleuth-eval
description: This benchmark evaluates Large Audio-Language Models (LALMs) on their ability to detect, localize, and discriminate speaker inconsistencies in multi-turn dialogues. It specifically probes whether models rely on acoustic cues or are biased toward textual coherence when judging speaker consistency. Use when the user wants to benchmark on SpeakerSleuth, or asks about evaluating this task. Reports detection_accuracy, discrimination_accuracy, localization_f1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.04029
  bibtex_key: lee2026speakersleuth
  confidence: high
---

# speakersleuth-eval

> SpeakerSleuth: Evaluating Large Audio-Language Models as Judges for Multi-turn Speaker Consistency — Lee et al. (2026) (arXiv:2601.04029, 2026)

## What this evaluates

This benchmark evaluates Large Audio-Language Models (LALMs) on their ability to detect, localize, and discriminate speaker inconsistencies in multi-turn dialogues. It specifically probes whether models rely on acoustic cues or are biased toward textual coherence when judging speaker consistency.

## Datasets

- **SpeakerSleuth** — total ?; splits: test (-1)

## Metrics

- `detection_accuracy` **(primary)** — range: [0, 1]
  - Percentage of dialogues correctly classified as consistent or inconsistent.
- `discrimination_accuracy` **(primary)** — range: [0, 1]
  - Percentage of inconsistent dialogues where the correct inconsistent speaker is identified.
- `localization_f1` **(primary)** — range: [0, 1]
  - F1-score for correctly identifying the specific turn(s) containing the speaker inconsistency.

## Input / output format

**Input**: Audio-only input consisting of a multi-turn dialogue, plus a ≥3-second reference audio sample of the target speaker. In ablation settings, textual context of the dialogue may be appended, or the reference audio may be removed.

**Output**: Model must output a judgment for each dialogue: (1) consistency label (consistent/inconsistent), (2) turn index/indices where inconsistency occurs (for localization), and (3) identity of the inconsistent speaker (for discrimination).

## Scoring recipe

```python
def compute_metrics(preds, golds):
    # Detection & Discrimination Accuracy
    det_correct = sum(1 for p, g in zip(preds['det'], golds['det']) if p == g)
    det_acc = det_correct / len(golds['det'])
    
    disc_correct = sum(1 for p, g in zip(preds['disc'], golds['disc']) if p == g)
    disc_acc = disc_correct / len(golds['disc'])
    
    # Localization F1 (turn-level exact match)
    tp = sum(1 for p, g in zip(preds['loc'], golds['loc']) if set(p) == set(g))
    fp = sum(1 for p, g in zip(preds['loc'], golds['loc']) if set(p) != set(g) and len(p) > 0)
    fn = sum(1 for p, g in zip(preds['loc'], golds['loc']) if set(p) != set(g) and len(g) > 0)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    loc_f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return det_acc, disc_acc, loc_f1
```

## Common pitfalls

- Models exhibit strong modality bias, often prioritizing textual coherence over acoustic cues, which artificially inflates accuracy when text context is provided.
- Performance is highly sensitive to the presence of reference audio; removing it causes drastic accuracy drops or unpredictable fluctuations across models.
- Evaluation scenarios (S1, S2, S3) vary significantly in difficulty and structure, leading to non-comparable baseline scores if not reported per scenario.

## Evidence (verbatim from paper)

> We report accuracy scores for Detection and Discrimination, and F1-scores for Localization as primary metrics. Detailed metric computation methods are provided in Appendix[B.3], along with additional metrics (Precision, Recall, Exact-Match).

## Citation

```bibtex
@misc{lee2026speakersleuth,
  title={SpeakerSleuth: Evaluating Large Audio-Language Models as Judges for Multi-turn Speaker Consistency},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2601.04029}
}
```

- arXiv: 2601.04029