Robonar Eval

---
name: robonar-eval
description: Evaluates a multimodal robot narration framework's ability to select key events, generate natural language summaries, and perform failure analysis (risk estimation, localization, explanation, recovery) on real-world household robot tasks. Use when the user wants to benchmark on RoboNar, or asks about evaluating this task. Reports Accuracy on failure analysis tasks.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.12960
  bibtex_key: wang2024robonar
  confidence: high
---

# robonar-eval

> I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences — Wang et al. (2024) (arXiv:2411.12960, 2024)

## What this evaluates

Evaluates a multimodal robot narration framework's ability to select key events, generate natural language summaries, and perform failure analysis (risk estimation, localization, explanation, recovery) on real-world household robot tasks.

## Datasets

- **RoboNar** — total 70; splits: full (70)

## Metrics

- `Accuracy on failure analysis tasks` **(primary)** — range: [0, 1]
  - Calculated as the proportion of correctly predicted outcomes across four sub-tasks (Risk Estimation, Failure Localization, Failure Explanation, Recovery Recommendation) out of the total failure cases.
- `narration_quality_likert` — range: [1, 5]
  - Mean score on a 1-5 Likert scale across four dimensions: Naturalness, Informativeness, Coherence, and Overall Quality, rated by human participants.
- `failure_capture_rate` — range: [0, 1]
  - Ratio of successfully captured failure cases to the total number of failure cases in the dataset, evaluated across different sampling thresholds and modalities.

## Input / output format

**Input**: Synchronized multimodal sensor streams (RGB-D video from two cameras, joint/base readings, state information, diagnostics) and task planning history, downsampled into aligned keyframes.

**Output**: Natural language narrations summarizing robot experiences, and structured failure analysis predictions (risk status, failure timestamp, failure reason, recovery recommendation).

## Scoring recipe

```python
def score_failure_analysis(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if (pred['risk'] == gold_item['risk'] and
            abs(pred['timestamp'] - gold_item['timestamp']) < 1.0 and
            pred['reason'] == gold_item['reason'] and
            pred['recovery'] == gold_item['recovery']):
            correct += 1
    return correct / len(gold)

def score_narration_quality(ratings):
    return {dim: sum(r[dim]) / len(r) for dim in ['Naturalness', 'Informativeness', 'Coherence', 'Overall']}
```

## Common pitfalls

- Feeding raw, unsummarized multimodal sensor data directly to LLMs/VLMs yields significantly lower accuracy than using intermediate key-event summarization.
- Relying solely on visual inputs for failure analysis is insufficient; internal robot state and planning data are critical for accurate localization and explanation.
- User study latency measurements must be carefully timed per task, as raw data interfaces may appear informative but do not improve accuracy over video-only baselines.

## Evidence (verbatim from paper)

> We decompose the failurle analysis into four specific tasks: 1) Risk Estimation (Pred): if the method can identify risk before failure happens; 2) Failure Localization (Loc): if the method can identify the failure time ; 3) Failure Explanation (Exp): if the method can tell the failure reason; and 4) Recovery Recommendation (Rec): if the method can give reasonable recovery recommendations. Results are shown in Figure 5: Accuracy on failure analysis tasks using different methods.

## Citation

```bibtex
@misc{wang2024robonar,
  title={I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2411.12960}
}
```

- arXiv: 2411.12960