Sports Qa Eval

---
name: sports-qa-eval
description: Probes video question answering capabilities, specifically focusing on temporal reasoning, action causality, counterfactual inference, and fine-grained motion understanding within professional sports contexts. Use when the user wants to benchmark on Sports-QA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.01505
  bibtex_key: li2024sportsqa
  confidence: high
---

# sports-qa-eval

> Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports — Li et al. (2024) (arXiv:2401.01505, 2024)

## What this evaluates

Probes video question answering capabilities, specifically focusing on temporal reasoning, action causality, counterfactual inference, and fine-grained motion understanding within professional sports contexts.

## Datasets

- **Sports-QA** — total 94000; splits: train (-1), val (-1), test (-1); repo https://github.com/HopLee6/Sports-QA

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answer classes out of the total number of questions.
- `F1-score` — range: percent
  - Harmonic mean of precision and recall, computed to account for the long-tail distribution and class imbalance across the 191 answer categories.

## Input / output format

**Input**: Down-sampled video frames (5 FPS) with pre-extracted global/local appearance and motion features, paired with a natural language question.

**Output**: A single answer string selected from a fixed vocabulary of 191 classes.

## Scoring recipe

```python
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
accuracy = (correct / len(predictions)) * 100
# Macro-averaged F1 to handle class imbalance
f1 = f1_score(gold, predictions, average='macro') * 100
```

## Common pitfalls

- The dataset contains 191 answer classes with a long-tail distribution, so accuracy alone can be misleading; F1-score is required to properly evaluate performance on rare classes.
- Baseline models like BlindQA achieve high accuracy by exploiting question-answer semantic correlations without using visual features, making it crucial to verify visual grounding.
- Visual features are pre-extracted using Faster-RCNN and I3D rather than raw video inputs, which may limit direct comparison with end-to-end video-language models.

## Evidence (verbatim from paper)

> For evaluation metrics, we employ accuracy to showcase the models' performance on our dataset. Additionally, recognizing the long-tail distribution of our dataset, we evaluate methods using F1-score. F1-score is particularly important for sports VideoQA as it explicitly reflects performance averaged over imbalanced classes.

## Citation

```bibtex
@misc{li2024sportsqa,
  title={Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2401.01505}
}
```

- arXiv: 2401.01505