Botfails Eval

---
name: botfails-eval
description: This evaluation probes a model's ability to detect and classify robotic failures in real-world manipulation tasks. It specifically tests whether a system can distinguish between genuine task-disrupting failures and benign environmental deviations using multimodal observations and nominal demonstrations. Use when the user wants to benchmark on BotFails, Real-π dataset, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13788
  bibtex_key: rolland2026failureidentification
  confidence: medium
---

# botfails-eval

> Failure Identification in Imitation Learning Via Statistical and Semantic Filtering — Rolland et al. (2026) (arXiv:2604.13788, 2026)

## What this evaluates

This evaluation probes a model's ability to detect and classify robotic failures in real-world manipulation tasks. It specifically tests whether a system can distinguish between genuine task-disrupting failures and benign environmental deviations using multimodal observations and nominal demonstrations.

## Datasets

- **BotFails** — total ?; splits: train (-1), test (-1)
- **Real-π dataset** — total ?; splits: train (-1), test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive and false positive rates across all classification thresholds.
- `failure-detection accuracy` — range: [0, 1]
  - Percentage of correctly identified failure instances out of the total number of failure instances in the test set.

## Input / output format

**Input**: Multimodal observations including camera images, proprioceptive state, and natural language task instructions, aligned against compact representations of nominal demonstrations.

**Output**: Continuous anomaly scores and spatial heatmaps, followed by a binary classification (failure vs. benign deviation) after vision-language model filtering.

## Scoring recipe

```python
def compute_metrics(predictions, labels):
    fpr, tpr, _ = roc_curve(labels, predictions)
    auroc = auc(fpr, tpr)
    preds_binary = (predictions >= 0.5).astype(int)
    accuracy = mean(preds_binary == labels)
    return {'AUROC': auroc, 'failure-detection accuracy': accuracy}
```

## Common pitfalls

- The dataset lacks publicly available train/val/test splits and exact size, making direct replication difficult.
- Baseline methods are adapted to the experimental setting rather than used out-of-the-box, which may introduce implementation bias.
- Distinguishing benign deviations (e.g., background shifts) from genuine failures requires careful thresholding and semantic filtering, which can be sensitive to VLM prompt design.

## Evidence (verbatim from paper)

> Evaluated on the novel BotFails dataset, FIDeL achieves +5.30% AUROC and +17.38% failure-detection accuracy over baselines, demonstrating superior real-world applicability and interpretability.

## Citation

```bibtex
@misc{rolland2026failureidentification,
  title={Failure Identification in Imitation Learning Via Statistical and Semantic Filtering},
  author={Rolland et al. (2026)},
  year={2026},
  note={arXiv:2604.13788}
}
```

- arXiv: 2604.13788