Biasinear Eval

---
name: biasinear-eval
description: Evaluates the robustness and sensitivity of multimodal large language models (MLLMs) to perturbations in spoken multiple-choice questions. It probes how models handle variations in language, accent, speaker gender, and answer option ordering, measuring both absolute correctness and prediction stability across conditions. Use when the user wants to benchmark on BiasInEar, or asks about evaluating this task. Reports Question Entropy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.01030
  bibtex_key: [REDACTED]
  confidence: high
---

# biasinear-eval

> Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations — Wei et al. (2026) (arXiv:2602.01030, 2026)

## What this evaluates

Evaluates the robustness and sensitivity of multimodal large language models (MLLMs) to perturbations in spoken multiple-choice questions. It probes how models handle variations in language, accent, speaker gender, and answer option ordering, measuring both absolute correctness and prediction stability across conditions.

## Datasets

- **BiasInEar** — total ?; splits: test (-1); repo https://github.com/ntunlplab/BiasInEar

## Metrics

- `Question Entropy` **(primary)** — range: [0, 1]
  - Shannon entropy of the model's answer distribution across four options, normalized by base 4 to ensure values fall in [0, 1]. Formula: H_q = -sum_{o in {A,B,C,D}} p_q(o) log_4 p_q(o).
- `APES` — range: [0, 1]
  - Average Pairwise Entropy Shift quantifies entropy variation across levels of a perturbation variable (e.g., gender or accent). Formula: APES_q^v = (2/(L(L-1))) * sum_{i<j} |H_q^{l_i} - H_q^{l_j}|.
- `Fleiss’ Kappa` — range: [-1, 1]
  - Measures categorical agreement across variable perturbations while correcting for chance. Formula: kappa = (P_bar - P_e) / (1 - P_e), where P_bar is average observed agreement and P_e is expected agreement.

## Input / output format

**Input**: Concatenated audio segments containing a spoken question followed by four answer options (A, B, C, D), presented in either canonical or reversed order.

**Output**: A single selected option label (A, B, C, or D). The paper applies post-processing to correct formatting errors before scoring.

## Scoring recipe

```python
def score(predictions, gold, perturbations):
    # predictions: list of model outputs per perturbation condition
    # gold: ground truth option
    # perturbations: dict mapping variable -> list of levels
    
    # 1. Question Entropy (from model probs or one-hot predictions)
    H_q = -sum(p * log4(p) for p in probs)
    
    # 2. APES across variable levels
    H_levels = [compute_entropy(preds_for_level) for level in levels]
    APES = mean(abs(H_levels[i] - H_levels[j]) for i, j in pairs)
    
    # 3. Fleiss' Kappa across perturbations
    observed = agreement_rate(predictions)
    expected = sum(p**2 for p in overall_probs)
    kappa = (observed - expected) / (1 - expected)
    
    return H_q, APES, kappa
```

## Common pitfalls

- Models require temperature=0 and specific post-processing to strip formatting artifacts before robustness analysis.
- Option order reversal changes the mapping of labels to content, so shifts must be computed on the underlying options, not the fixed A/B/C/D labels.
- Audio segments must be correctly concatenated according to the experimental condition before model inference.

## Evidence (verbatim from paper)

> To evaluate robustness under input perturbations, we employ three complementary metrics: entropy, APES, and Fleiss’ Kappa. These measures go beyond accuracy by assessing not only correctness but also the stability and consistency of model behavior.

## Citation

```bibtex
@misc{wei2026biasinear,
  title={Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations},
  author={Wei et al. (2026)},
  year={2026},
  note={arXiv:2602.01030}
}
```

- arXiv: 2602.01030