Robocse Eval

---
name: robocse-eval
description: Evaluates a robot's ability to generalize semantic knowledge by predicting object affordances, locations, and materials in unseen environments and inferring ranks of unseen semantic triples. It probes multi-relational embedding performance for common-sense reasoning in residential robotics. Use when the user wants to benchmark on AI2Thor, or asks about evaluating this task. Reports MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1903.00412
  bibtex_key: daruna2019robocse
  confidence: high
---

# robocse-eval

> RoboCSE: Robot Common Sense Embedding — Daruna et al. (2019) (arXiv:1903.00412, 2019)

## What this evaluates

Evaluates a robot's ability to generalize semantic knowledge by predicting object affordances, locations, and materials in unseen environments and inferring ranks of unseen semantic triples. It probes multi-relational embedding performance for common-sense reasoning in residential robotics.

## Datasets

- **AI2Thor** — total 15000; splits: train (-1), val (-1), test (-1)

## Metrics

- `MRR` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank: average of 1/rank for each query, where rank is determined by the number of observations in the ground truth distribution.
- `Hits@K` — range: [0, 1]
  - Percentage of queries where the correct triple appears in the top K ranked predictions.

## Input / output format

**Input**: Semantic triples (head, relation, tail) from AI2Thor, where the model must predict the missing element given the other two.

**Output**: Ranked list of candidate triples or entities, ordered by predicted likelihood/rank.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, K=10):
    reciprocal_ranks = []
    hits_at_k = []
    for pred_list, gt in zip(predictions, ground_truth):
        rank = pred_list.index(gt) + 1
        reciprocal_ranks.append(1.0 / rank)
        hits_at_k.append(1.0 if rank <= K else 0.0)
    mrr = sum(reciprocal_ranks) / len(reciprocal_ranks)
    hits = sum(hits_at_k) / len(hits_at_k)
    return mrr, hits
```

## Common pitfalls

- Negative sampling requires filtering perturbed triples to ensure they are not already in the training set; naive perturbation yields worse results.
- Ground truth ranking is based on observation frequency rather than binary truth values, which differs from standard link prediction benchmarks.
- For environment generalization, splits must be performed at the environment level (not triple level) to prevent data leakage across folds.

## Evidence (verbatim from paper)

> Error metrics similar to those from the relational embedding community (MRR* and Hits@K*) were calculated using the ground truth rank for comparison. The standard MRR and Hits@K were used to measure the algorithm’s performance, allowing us to assess how frequently the robot was correct on the first attempt.

## Citation

```bibtex
@misc{daruna2019robocse,
  title={RoboCSE: Robot Common Sense Embedding},
  author={Daruna et al. (2019)},
  year={2019},
  note={arXiv:1903.00412}
}
```

- arXiv: 1903.00412