Cameramotion Vqa Eval

---
name: cameramotion-vqa-eval
description: Evaluates fine-grained camera motion recognition in VideoLLMs using multiple-choice questions and multi-label classification. Probes whether models can distinguish geometric camera movements from object motion and static shots. Use when the user wants to benchmark on CameraMotionVQA, CameraMotionDataset, or asks about evaluating this task. Reports answer accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.13119
  bibtex_key: [REDACTED]
  confidence: high
---

# cameramotion-vqa-eval

> Geometry-Guided Camera Motion Understanding in VideoLLMs — Feng et al. (2026) (arXiv:2603.13119, 2026)

## What this evaluates

Evaluates fine-grained camera motion recognition in VideoLLMs using multiple-choice questions and multi-label classification. Probes whether models can distinguish geometric camera movements from object motion and static shots.

## Datasets

- **CameraMotionVQA** — total ?; splits: test (-1)
- **CameraMotionDataset** — total ?; splits: test (-1)

## Metrics

- `answer accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted options in the 4-way multiple-choice setting.
- `instance accuracy` — range: [0, 1]
  - Exact match accuracy: 1 if all predicted labels match the ground truth labels for a sample, else 0.
- `Macro-F1` — range: [0, 1]
  - Unweighted mean of F1 scores computed independently for each motion primitive class.
- `Weighted-F1` — range: [0, 1]
  - F1 score averaged across classes, weighted by the number of true instances for each class.

## Input / output format

**Input**: Video segment/clips paired with a 4-way multiple-choice question prompt (VQA) or raw video frames for multi-label classification.

**Output**: Single selected option (VQA) or a multi-label vector over predefined camera motion primitives (classification).

## Scoring recipe

```python
def compute_metrics(preds, golds):
    # VQA / Instance Accuracy
    correct = sum(1 for p, g in zip(preds, golds) if p == g)
    acc = correct / len(golds)
    
    # Multi-label Instance Accuracy
    inst_acc = sum(1 for p, g in zip(preds, golds) if np.array_equal(p, g)) / len(golds)
    
    # Macro & Weighted F1
    macro_f1 = f1_score(golds, preds, average='macro', zero_division=0)
    weighted_f1 = f1_score(golds, preds, average='weighted', zero_division=0)
    return acc, inst_acc, macro_f1, weighted_f1
```

## Common pitfalls

- Models frequently confuse geometrically similar primitives (e.g., truck vs. pan).
- Salient object motion often causes inconsistent directional predictions (e.g., pan-left vs. pan-right).
- Static segments are out-of-distribution for 3D foundation models, leading to unreliable predictions.

## Evidence (verbatim from paper)

> CameraMotionVQA formats each segment as a 4-way multiple-choice question, and models are evaluated using answer accuracy. For evaluating the camera motion multi-label task, we report instance accuracy (exact match of all labels), Macro-F1 across motion primitives, and Weighted-F1 weighted by label frequency.

## Citation

```bibtex
@misc{feng2026geometryguided,
  title={Geometry-Guided Camera Motion Understanding in VideoLLMs},
  author={Feng et al. (2026)},
  year={2026},
  note={arXiv:2603.13119}
}
```

- arXiv: 2603.13119