Meccano Eval

---
name: meccano-eval
description: Evaluates multimodal egocentric video understanding in an industrial-like setting. It probes action recognition, active object detection, human-object interaction, and future action anticipation using synchronized RGB, depth, and gaze signals. Use when the user wants to benchmark on MECCANO, or asks about evaluating this task. Reports Top-1 Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.08691
  bibtex_key: ragusa2022meccano
  confidence: high
---

# meccano-eval

> MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain — Ragusa et al. (2022) (arXiv:2209.08691, 2022)

## What this evaluates

Evaluates multimodal egocentric video understanding in an industrial-like setting. It probes action recognition, active object detection, human-object interaction, and future action anticipation using synchronized RGB, depth, and gaze signals.

## Datasets

- **MECCANO** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Top-1 Accuracy` **(primary)** — range: [0, 1]
  - Percentage of video segments where the predicted action class exactly matches the ground truth label.
- `AVG F1-score` — range: [0, 1]
  - Class-mean F1-score computed across all action classes, averaging precision and recall per class before macro-averaging.
- `AP (IoU>0.5)` — range: [0, 1]
  - Average Precision for active object detection using a 0.5 Intersection over Union threshold, following standard Pascal VOC mAP conventions.

## Input / output format

**Input**: Synchronized RGB frames, depth maps, and gaze fixation maps from egocentric videos.

**Output**: Action class labels for temporal segments, bounding boxes with class labels for active objects, and interaction tuples for EHOIs.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    # Top-1 Accuracy
    top1 = sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)
    
    # AP (IoU>0.5)
    ap = compute_ap(preds, gold, iou_threshold=0.5)
    
    # AVG F1-score
    f1s = []
    for cls in set(gold):
        tp = sum(1 for p, g in zip(preds, gold) if p == g == cls)
        fp = sum(1 for p in preds if p == cls and p not in gold)
        fn = sum(1 for g in gold if g == cls and g not in preds)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    avg_f1 = sum(f1s) / len(f1s)
    return {'Top-1 Accuracy': top1, 'AP (IoU>0.5)': ap, 'AVG F1-score': avg_f1}
```

## Common pitfalls

- Gaze modality yields only marginal gains over RGB-Depth fusion due to the simplicity of the baseline architecture.
- Standard hand-object detectors fail to generalize without domain-specific retraining on the industrial dataset.
- Active objects can outnumber hands (up to 7 per frame), requiring relaxed distance thresholds for detection baselines.

## Evidence (verbatim from paper)

> We evaluate action recognition using Top-1 and Top-5 accuracy computed on the whole test set. As class-aware measures, we report class-mean precision, class-mean recall and F1-score.

## Citation

```bibtex
@misc{ragusa2022meccano,
  title={MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain},
  author={Ragusa et al. (2022)},
  year={2022},
  note={arXiv:2209.08691}
}
```

- arXiv: 2209.08691