Interactive Audio Eval

---
name: interactive-audio-eval
description: Evaluates Large Audio Models (LAMs) on real-world, task-oriented voice assistant interactions by capturing user preferences through open-ended pairwise comparisons. It measures how well models align with actual user needs and preferences in an interactive setting, rather than relying on static reference-based benchmarks. Use when the user wants to benchmark on TalkArena Interactive User Preferences, or asks about evaluating this task. Reports Bradley-Terry model score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.15919
  bibtex_key: [REDACTED]
  confidence: high
---

# interactive-audio-eval

> Mind the Gap! Static and Interactive Evaluations of Large Audio Models — Li et al. (2025) (arXiv:2502.15919, 2025)

## What this evaluates

Evaluates Large Audio Models (LAMs) on real-world, task-oriented voice assistant interactions by capturing user preferences through open-ended pairwise comparisons. It measures how well models align with actual user needs and preferences in an interactive setting, rather than relying on static reference-based benchmarks.

## Datasets

- **TalkArena Interactive User Preferences** — total 7500; splits: full (7500)

## Metrics

- `Bradley-Terry model score` **(primary)** — range: other
  - A latent preference score derived from pairwise user votes. The probability that model i is preferred over model j is Pr(i>j) = e^{β_i} / (e^{β_i} + e^{β_j}), where β coefficients are optimized by maximizing the log-likelihood of observed preferences. Final scores are computed as p_i = e^{β_i}.

## Input / output format

**Input**: User voice query (streamed character-by-character) + two anonymous model responses presented in random order.

**Output**: User selects preferred model (i > j, j > i, or tie/no preference) + optional text or speech justification.

## Scoring recipe

```python
D = list of (model_i, model_j, y) where y ∈ {1, 0.5, 0}
# Optimize β to maximize log-likelihood
L(β) = Σ [y*log(σ(β_i - β_j)) + (1-y)*log(σ(β_j - β_i))]
β_opt = minimize(-L(β), method='L-BFGS')
scores = {m: exp(β_opt[m]) for m in models}
rank = sorted(models, key=lambda m: scores[m], reverse=True)
```

## Common pitfalls

- Pairwise votes yield relative rankings, not absolute performance scores, making cross-session or cross-pool comparisons invalid without re-estimating the Bradley-Terry coefficients.
- The Bradley-Terry model assumes transitivity of preferences, which may not hold in open-ended, style-driven user interactions.
- Positional and personal biases are mitigated by random ordering and anonymization, but users may still prefer models based on response latency or conversational style rather than factual accuracy.

## Evidence (verbatim from paper)

> After submitting a query, users receive responses from two anonymous models, which are randomly selected and ordered in order to avoid personal and positional bias in their preferences. For assessment, users provide a simple pairwise preference ranking—choosing the better response or indicating no preference between the two. To convert the collected pairwise preference data to model ranking, we apply the Bradley-Terry model ... The Bradley-Terry coefficients β are computed by maximizing the log-likelihood of the observed pairwise preferences D ... We then compute p_i = e^{β_i}.

## Citation

```bibtex
@misc{li2025mindthegap,
  title={Mind the Gap! Static and Interactive Evaluations of Large Audio Models},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2502.15919}
}
```

- arXiv: 2502.15919