Matt Attribution Eval

---
name: matt-attribution-eval
description: This benchmark evaluates fine-grained mistake understanding in egocentric videos by attributing errors to specific semantic roles, temporal points of no return, and spatial locations. It probes a model's ability to align video content with instructional text, localize manipulation events, and classify whether actions deviate from intended goals. Use when the user wants to benchmark on Ego4D-M, EPIC-KITCHENS-M, EgoPER, or asks about evaluating this task. Reports F1@0.5, MAE, mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.20525
  bibtex_key: li2025mistakeattribution
  confidence: high
---

# matt-attribution-eval

> Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos — Yayuan Li et al. (2025) (arXiv:2511.20525, 2025)

## What this evaluates

This benchmark evaluates fine-grained mistake understanding in egocentric videos by attributing errors to specific semantic roles, temporal points of no return, and spatial locations. It probes a model's ability to align video content with instructional text, localize manipulation events, and classify whether actions deviate from intended goals.

## Datasets

- **Ego4D-M** — total 256000; splits: train (206000), val (25000), test (25000)
- **EPIC-KITCHENS-M** — total 220000; splits: train (176000), val (22000), test (22000)
- **EgoPER** — total ?; splits: test (-1)

## Metrics

- `F1@0.5` **(primary)** — range: [0, 1]
  - Binary classification F1 score computed at a 0.5 confidence threshold. Calculated as 2*TP/(2*TP+FP+FN) per semantic role or averaged across roles.
- `Accuracy` — range: [0, 1]
  - Percentage of correctly classified instances (mistake vs. correct) per role or averaged.
- `MAE` **(primary)** — range: other
  - Mean Absolute Error between predicted and ground-truth Point-of-No-Return frame index or time duration.
- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection-over-Union between predicted and ground-truth bounding boxes for the manipulated object/hands in the PNR frame.
- `Center Distance (CD)` — range: other
  - Normalized Euclidean distance between the centers of predicted and ground-truth bounding boxes.
- `Box Size Error (BSE)` — range: other
  - Normalized difference between the width and height dimensions of predicted and ground-truth bounding boxes.

## Input / output format

**Input**: Egocentric video clips paired with corresponding instructional text. For baselines, structured prompts are used to elicit binary decisions per semantic role or localization outputs.

**Output**: Binary classification per semantic role (mistake/correct), predicted frame index or time for temporal localization, bounding box coordinates for spatial localization, and binary clip-level mistake label for detection.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    # Semantic/Detection: F1@0.5 & Accuracy
    y_pred = [1 if p > 0.5 else 0 for p in preds]
    acc = sum(y_pred == gold) / len(gold)
    tp = sum(1 for t, p in zip(gold, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(gold, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(gold, y_pred) if t == 1 and p == 0)
    f1 = 2*tp / (2*tp + fp + fn) if (2*tp + fp + fn) > 0 else 0
    # Temporal: MAE (frames/seconds)
    mae = sum(abs(p - g) for p, g in zip(preds_frames, gold_frames)) / len(gold_frames)
    # Spatial: mIoU, CD, BSE
    miou = mean(iou(p_box, g_box) for p_box, g_box in zip(preds_boxes, gold_boxes))
    cd = mean(box_center_dist(p_box, g_box) for p_box, g_box in zip(preds_boxes, gold_boxes))
    bse = mean(box_size_error(p_box, g_box) for p_box, g_box in zip(preds_boxes, gold_boxes))
    return {'F1@0.5': f1, 'Accuracy': acc, 'MAE': mae, 'mIoU': miou, 'CD': cd, 'BSE': bse}
```

## Common pitfalls

- Mistake detection baselines are designed for out-of-distribution detection on a small set of activities and fail on large-scale benchmarks because they cannot separate actual mistakes from benign variability across many activities.
- Standard hand-detection models (e.g., MediaPipe) cannot be used for spatial attribution because they do not localize the manipulated object, which is required for precise mistake grounding.
- Feeding raw text embeddings directly into attribution heads without a projection block significantly degrades semantic attribution performance.

## Evidence (verbatim from paper)

> For semantic attribution, we treat each semantic role as a binary classification problem. Following prior work[[23]], we report F1@0.5 and Accuracy as the main metrics, both per semantic role and averaged across roles. Following the PNR localization task[[12]], we evaluate temporal attribution by Mean Absolute Error, both per frame and per second. For spatial attribution, we report mean Intersection-over-Union (mIoU) between the predicted and ground-truth boxes, and additionally report Center Distance (CD) and Box Size Error (BSE).

## Citation

```bibtex
@misc{li2025mistakeattribution,
  title={Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos},
  author={Yayuan Li et al. (2025)},
  year={2025},
  note={arXiv:2511.20525}
}
```

- arXiv: 2511.20525