Videgothink Eval

---
name: videgothink-eval
description: Egocentric video understanding for embodied AI, probing capabilities in video question-answering, hierarchical task planning, visual grounding, and reward modeling. It evaluates how well multimodal models comprehend first-person, action-oriented video contexts required for robotic interaction. Use when the user wants to benchmark on VidEgoThink, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.11623
  bibtex_key: cheng2024videgothink
  confidence: high
---

# videgothink-eval

> VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI — Cheng et al. (2024) (arXiv:2410.11623, 2024)

## What this evaluates

Egocentric video understanding for embodied AI, probing capabilities in video question-answering, hierarchical task planning, visual grounding, and reward modeling. It evaluates how well multimodal models comprehend first-person, action-oriented video contexts required for robotic interaction.

## Datasets

- **VidEgoThink** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers, labels, or grounded regions compared to ground truth annotations. Computed per sub-task and averaged across object, action, and scene dimensions for QA, and across planning, grounding, and reward modeling tasks.

## Input / output format

**Input**: Egocentric video clips (or sampled keyframes/captions for API models) paired with task-specific questions or instructions.

**Output**: Natural language responses (for QA, planning, feedback), bounding boxes or temporal segments (for visual grounding), or binary classifications (for critique).

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if normalize_text(pred) == gold:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- GPT-4o's privacy filters trigger refusals on indoor egocentric videos, causing performance drops when more frames are provided rather than improving them.
- Open-source MLLMs frequently fail to follow strict output formats for hierarchy planning, leading to artificially low accuracy scores due to format mismatches.
- Visual grounding tasks use novel formats not optimized for existing MLLMs, making direct performance comparisons with specialized grounding models less straightforward.

## Evidence (verbatim from paper)

> MLLMs perform poorly, with a best average accuracy of 32.82% across all dimensions (35.00% for object, 28.33% for action, and 26.33% for scene elements), indicating struggles with egocentric video question-answering.

## Citation

```bibtex
@misc{cheng2024videgothink,
  title={VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2410.11623}
}
```

- arXiv: 2410.11623