Handy Vqa Eval

---
name: handy-vqa-eval
description: Evaluates video foundation models' ability to understand fine-grained spatiotemporal dynamics in hand-object interactions. It probes spatial reasoning, motion tracking, and part-level geometric grounding through multiple-choice questions and video object segmentation tasks. Use when the user wants to benchmark on HanDyVQA, or asks about evaluating this task. Reports top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.00885
  bibtex_key: tateno2025handyvqa
  confidence: high
---

# handy-vqa-eval

> HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics — Tateno et al. (2025) (arXiv:2512.00885, 2025)

## What this evaluates

Evaluates video foundation models' ability to understand fine-grained spatiotemporal dynamics in hand-object interactions. It probes spatial reasoning, motion tracking, and part-level geometric grounding through multiple-choice questions and video object segmentation tasks.

## Datasets

- **HanDyVQA** — total 11100; splits: test (-1)

## Metrics

- `top-1 accuracy` **(primary)** — range: [0, 1]
  - Fraction of questions where the model's predicted option matches the ground-truth answer. Averaged across Action, Process, Location, State, and Parts categories.
- `Average Precision (AP)` — range: [0, 1]
  - Area under the precision-recall curve for the Objects category, used because questions can have multiple correct answers.
- `Jaccard Index (J)` — range: [0, 1]
  - Intersection over union between predicted and ground-truth segmentation masks, averaged over all annotated frames in a video.
- `Boundary F-measure (F)` — range: [0, 1]
  - F-score computed between the boundaries of predicted and ground-truth masks, averaged over all annotated frames.

## Input / output format

**Input**: Video clips (uniformly sampled to 16 frames) paired with a natural language question and a list of multiple-choice options. For ReasoningVOS, the input is the video plus either the question or the ground-truth option as a text prompt.

**Output**: For MCQ: a single selected option (letter or text). For ReasoningVOS: per-frame binary/instance segmentation masks corresponding to the queried hand, object, or object part.

## Scoring recipe

```python
# MCQ Scoring
correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
accuracy = correct / len(golds)

# Objects AP Scoring
ap = compute_average_precision(gold_masks, pred_masks)

# ReasoningVOS Scoring
j_scores = []
f_scores = []
for frame in frames:
    j_scores.append(jaccard_index(pred_mask[frame], gold_mask[frame]))
    f_scores.append(boundary_f_measure(pred_mask[frame], gold_mask[frame]))
j_acc = mean(j_scores)
f_acc = mean(f_scores)
```

## Common pitfalls

- Models often rely on superficial textual or contextual cues rather than actual video dynamics, leading to inflated scores on text-only baselines if not carefully controlled.
- The Objects category uses Average Precision instead of accuracy, which can cause confusion when computing the overall average score across categories.
- Error rates are computed per error type based on distractor analysis, but a single wrong answer may correspond to multiple error types, complicating aggregate error reporting.

## Evidence (verbatim from paper)

> Evaluation metrics. We report top-1 accuracy for all the categories except Objects, and Average Precision (AP) for Objects because it has more than one answers per question. Following standard VOS evaluation protocols [30, 49], we use the Jaccard Index (I) and Boundary F-measure (F) computed for each frame and report their average over annotated frames.

## Citation

```bibtex
@misc{tateno2025handyvqa,
  title={HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics},
  author={Tateno et al. (2025)},
  year={2025},
  note={arXiv:2512.00885}
}
```

- arXiv: 2512.00885