Epic Kitchens 100 Mqa Eval

---
name: epic-kitchens-100-mqa-eval
description: Evaluates multi-modal large language models' ability to recognize and distinguish between similar human actions in egocentric videos through multiple-choice question answering. It specifically probes fine-grained action discrimination using hard, semantically and visually similar distractors generated by action recognition models. Use when the user wants to benchmark on EPIC-KITCHENS-100-MQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.18712
  bibtex_key: qi2025llavaction
  confidence: high
---

# epic-kitchens-100-mqa-eval

> LLaVAction: evaluating and training multi-modal large language models for action understanding — Qi et al. (2025) (arXiv:2503.18712, 2025)

## What this evaluates

Evaluates multi-modal large language models' ability to recognize and distinguish between similar human actions in egocentric videos through multiple-choice question answering. It specifically probes fine-grained action discrimination using hard, semantically and visually similar distractors generated by action recognition models.

## Datasets

- **EPIC-KITCHENS-100-MQA** — total ?; splits: train (-1), test (-1); repo https://github.com/AdaptiveMotorControlLab/LLaVAction

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted options out of the total number of multiple-choice questions. Calculated as (correct predictions / total questions) × 100.

## Input / output format

**Input**: Video clips represented as a fixed number of frames (typically 8 or 16), accompanied by a multiple-choice question with 5 options (1 ground truth narration + 4 hard distractors generated by TIM or AVION).

**Output**: Selection of exactly one option from the 5 provided choices.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Distractor generation method drastically changes task difficulty; TIM-generated hard negatives are significantly harder than random distractors.
- Performance is highly sensitive to the number of input frames (8 vs 16 vs 32 vs 64), so results must be compared at identical frame counts.
- The benchmark uses egocentric video narration, so models must handle viewpoint-specific action representations rather than third-person descriptions.

## Evidence (verbatim from paper)

> In our EPIC-KITCHENS-100-MQA benchmark, ground truth narration is within the 5 options. In addition to ground truth, the distractors are generated from TIM with its highest confidence predictions (see Section[3.1]). ... Percent accuracy is shown.

## Citation

```bibtex
@misc{qi2025llavaction,
  title={LLaVAction: evaluating and training multi-modal large language models for action understanding},
  author={Qi et al. (2025)},
  year={2025},
  note={arXiv:2503.18712}
}
```

- arXiv: 2503.18712