Incompebench Eval

---
name: incompebench-eval
description: This benchmark evaluates fine-grained music information retrieval by measuring how well models match audio tracks to diverse text queries. It probes the ability to capture nuanced musical attributes, handle negations, and rank candidates based on graded relevance rather than binary matches. Use when the user wants to benchmark on IncompeBench, or asks about evaluating this task. Reports graded relevance (0-3).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.11941
  bibtex_key: clavie2026incompebench
  confidence: high
---

# incompebench-eval

> IncompeBench: A Permissively Licensed, Fine-Grained Benchmark for Music Information Retrieval — Clavié et al. (2026) (arXiv:2602.11941, 2026)

## What this evaluates

This benchmark evaluates fine-grained music information retrieval by measuring how well models match audio tracks to diverse text queries. It probes the ability to capture nuanced musical attributes, handle negations, and rank candidates based on graded relevance rather than binary matches.

## Datasets

- **IncompeBench** — total 128000; splits: test (128000); repo https://github.com/mixedbread-ai/incompebench-programs

## Metrics

- `graded relevance (0-3)` **(primary)** — range: [0, 3]
  - A 4-point relevance scale where 0 = completely irrelevant, 1 = partially relevant, 2 = highly relevant, and 3 = fully relevant to every aspect of the query. Used to compute standard ranking metrics like nDCG and MRR.

## Input / output format

**Input**: Text query + 30-second audio chunk (16kHz) for each candidate song.

**Output**: Graded relevance score (integer 0, 1, 2, or 3).

## Scoring recipe

```python
def compute_metric(predictions, gold):
    # predictions: list of graded relevance scores (0-3) for retrieved candidates
    # gold: list of graded relevance scores (0-3) for the same candidates
    # Compute Mean Reciprocal Rank (MRR) based on fully relevant (3) items
    for i, pred in enumerate(predictions):
        if pred == 3:
            return 1.0 / (i + 1)
    return 0.0
```

## Common pitfalls

- Query negations (e.g., 'high BPM without guitar') require careful handling to avoid lenient matches.
- Candidate generation uses Reciprocal Rank Fusion across diverse models, which may bias ranking evaluation toward models used in the fusion step.
- LLM-generated annotations exhibit leniency; the benchmark releases 'Lenient' and 'Strict' variants to account for this noise.

## Evidence (verbatim from paper)

> The labelling process is conducted with Gemini 3 Pro... with fine-grained relevance levels, ranging from 0 (completely irrelevant) to 3 (fully relevant to every aspect of the query).

## Citation

```bibtex
@misc{clavie2026incompebench,
  title={IncompeBench: A Permissively Licensed, Fine-Grained Benchmark for Music Information Retrieval},
  author={Clavié et al. (2026)},
  year={2026},
  note={arXiv:2602.11941}
}
```

- arXiv: 2602.11941