Exact Eval

---
name: exact-eval
description: Evaluates video-language models' ability to answer questions about expert-level physical skilled activities in long-form videos. It probes fine-grained action recognition, temporal reasoning, and domain-specific generalization across sports, bike repair, cooking, health, music, and dance. Use when the user wants to benchmark on ExAct, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.06277
  bibtex_key: yi2025exact
  confidence: high
---

# exact-eval

> ExAct: A Video-Language Benchmark for Expert Action Analysis — Yi et al. (2025) (arXiv:2506.06277, 2025)

## What this evaluates

Evaluates video-language models' ability to answer questions about expert-level physical skilled activities in long-form videos. It probes fine-grained action recognition, temporal reasoning, and domain-specific generalization across sports, bike repair, cooking, health, music, and dance.

## Datasets

- **ExAct** — total 3521; splits: test (3521)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of questions for which the model selects the correct answer. Computed per activity and averaged across all 11 activities.

## Input / output format

**Input**: Video clips (uniformly sampled to 32 frames at 796x448 resolution) paired with a multiple-choice question about expert physical skilled activities.

**Output**: A selected answer choice from the provided options.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p == g)
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Frame extraction must use uniform sampling of exactly 32 frames at 796x448 resolution for all models to maintain fairness.
- Accuracy is aggregated by first computing per-activity/domain accuracy and then averaging across the 11 activities, rather than calculating a single global accuracy over all 3,521 samples.
- Models resize the 796x448 frames internally according to their own input requirements, which can introduce variability if not carefully controlled.

## Evidence (verbatim from paper)

> We use standard question-answering accuracy as the primary evaluation metric. Our benchmark includes a total of 3,521 QA samples spanning 11 fine-grained physical skilled activities. For each activity, we compute the accuracy as the percentage of questions for which the model selects the correct answer. To summarize performance across the dataset, we report the average accuracy across all 11 activities.

## Citation

```bibtex
@misc{yi2025exact,
  title={ExAct: A Video-Language Benchmark for Expert Action Analysis},
  author={Yi et al. (2025)},
  year={2025},
  note={arXiv:2506.06277}
}
```

- arXiv: 2506.06277