Refereebench Eval

---
name: refereebench-eval
description: Evaluates Multimodal Large Language Models (MLLMs) on automatic sports refereeing tasks, probing their ability to detect incidents, classify fouls, apply sport-specific rules, and ground decisions temporally across 11 different sports. Use when the user wants to benchmark on RefereeBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.15736
  bibtex_key: xu2026refereebench
  confidence: high
---

# refereebench-eval

> RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees — Xu et al. (2026) (arXiv:2604.15736, 2026)

## What this evaluates

Evaluates Multimodal Large Language Models (MLLMs) on automatic sports refereeing tasks, probing their ability to detect incidents, classify fouls, apply sport-specific rules, and ground decisions temporally across 11 different sports.

## Datasets

- **RefereeBench** — total 6475; splits: test (6475)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Computed by directly comparing the model’s predicted answer with the ground truth answer. A match yields 1, otherwise 0. Averaged over all QA pairs.

## Input / output format

**Input**: A full video clip (fixed at 720p resolution, model's default FPS) paired with a multiple-choice question describing a sports officiating scenario.

**Output**: The model's selected multiple-choice answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models tend to over-call fouls on negative samples (legal plays), showing high misidentification rates especially under suggestive wording.
- Performance varies significantly across sports, so a high overall score does not guarantee stable cross-sport generalization.
- Audio modality substantially impacts performance; evaluating with frames-only underestimates model capabilities compared to full video input.

## Evidence (verbatim from paper)

> The accuracy is computed by directly comparing the model’s output with the ground truth answer without the need for any external models.

## Citation

```bibtex
@misc{xu2026refereebench,
  title={RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees},
  author={Xu et al. (2026)},
  year={2026},
  note={arXiv:2604.15736}
}
```

- arXiv: 2604.15736