Timid Robot Mistake Detection Eval

---
name: timid-robot-mistake-detection-eval
description: This benchmark evaluates a model's ability to detect time-dependent and physical mistakes in robotic task executions from video. It specifically probes temporal reasoning, semantic task violation detection, and sim-to-real generalization by comparing frame-level anomaly predictions against ground-truth annotations. Use when the user wants to benchmark on BridgeData V2, Multi-robot dataset, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.09782
  bibtex_key: gallego2026timid
  confidence: high
---

# timid-robot-mistake-detection-eval

> TIMID: Time-Dependent Mistake Detection in Videos of Robot Executions — Gallego et al. (2026) (arXiv:2603.09782, 2026)

## What this evaluates

This benchmark evaluates a model's ability to detect time-dependent and physical mistakes in robotic task executions from video. It specifically probes temporal reasoning, semantic task violation detection, and sim-to-real generalization by comparing frame-level anomaly predictions against ground-truth annotations.

## Datasets

- **BridgeData V2** — total 1000; splits: train (800), test (200)
- **Multi-robot dataset** — total ?; splits: train (-1), test (-1)

## Metrics

- `Average Precision (AP)` — range: [0, 100] percent
  - Area under the precision-recall curve computed over frame-level predictions, summarizing detection performance across all thresholds.
- `Average Recall (AR)` — range: [0, 100] percent
  - Average recall across frames or videos, measuring the fraction of actual mistake frames correctly identified by the model.
- `F1` **(primary)** — range: [0, 100] percent
  - Harmonic mean of precision and recall at the frame level, balancing false positives and false negatives to report overall detection accuracy.

## Input / output format

**Input**: Video sequences of robot executions (processed as frames or frame batches), accompanied by task/mistake prompts specifying the expected behavior and error types.

**Output**: Frame-level binary predictions or anomaly scores indicating whether a mistake is occurring at each frame.

## Scoring recipe

```python
def compute_frame_metrics(predictions, gold):
    tp = sum(p == 1 and g == 1 for p, g in zip(predictions, gold))
    fp = sum(p == 1 and g == 0 for p, g in zip(predictions, gold))
    fn = sum(p == 0 and g == 1 for p, g in zip(predictions, gold))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    ap = compute_average_precision_curve(precision, recall)
    return ap, recall, f1
```

## Common pitfalls

- Models are trained with weak video-level supervision (correct vs. erroneous) but must generate frame-level predictions, creating a label alignment challenge that can inflate or deflate metrics depending on thresholding.
- Inference time varies drastically between baselines (e.g., VLMs take hours vs. seconds for TIMID), which is a critical deployment constraint often overlooked when comparing accuracy metrics.
- Sim-to-real domain shift causes significant performance drops across all architectures, so evaluating only on simulation data overestimates real-world applicability.

## Evidence (verbatim from paper)

> To evaluate the models, we measure standard detection metrics, Average Precision (AP), Average Recall (AR) and F1, computed at frame level.

## Citation

```bibtex
@misc{gallego2026timid,
  title={TIMID: Time-Dependent Mistake Detection in Videos of Robot Executions},
  author={Gallego et al. (2026)},
  year={2026},
  note={arXiv:2603.09782}
}
```

- arXiv: 2603.09782