Real Robot Challenge 2022 Eval

---
name: real-robot-challenge-2022-eval
description: Evaluates offline reinforcement learning and imitation learning algorithms on real-world dexterous manipulation tasks. It probes the ability to learn precise in-hand orientation and stable grasping from pre-collected robot data without online interaction, and measures transfer performance to physical hardware. Use when the user wants to benchmark on Real Robot Challenge 2022 TriFinger Datasets, or asks about evaluating this task. Reports overall score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.07741
  bibtex_key: [REDACTED]
  confidence: medium
---

# real-robot-challenge-2022-eval

> Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World — Gurtler et al. (2023) (arXiv:2308.07741, 2023)

## What this evaluates

Evaluates offline reinforcement learning and imitation learning algorithms on real-world dexterous manipulation tasks. It probes the ability to learn precise in-hand orientation and stable grasping from pre-collected robot data without online interaction, and measures transfer performance to physical hardware.

## Datasets

- **Real Robot Challenge 2022 TriFinger Datasets** — total ?; splits: Expert (-1), Mixed (-1)

## Metrics

- `overall score` **(primary)** — range: other
  - Average of the mean returns across four task/dataset combinations: Push/Expert, Push/Mixed, Lift/Expert, and Lift/Mixed. Failed execution runs are assigned a return of 0.
- `average return` — range: other
  - Mean reward accumulated per task/dataset combination over multiple evaluation runs. Standard error of the mean is also reported.

## Input / output format

**Input**: Pre-collected TriFinger robot datasets containing observations (e.g., images, proprioception) and actions for Push and Lift tasks.

**Output**: A learned behavior policy or offline RL algorithm executed on real hardware via remote access to generate actions for the robot.

## Scoring recipe

```python
def compute_overall_score(task_returns):
    # task_returns: dict mapping task/dataset combo to mean return
    # failed runs are already counted as 0 in the mean return
    overall = sum(task_returns.values()) / len(task_returns)
    return overall
```

## Common pitfalls

- Hyperparameters for offline RL algorithms are often tuned on simulated versions of the tasks, which can inflate real-world performance compared to purely real-world tuned methods.
- Failed physical execution runs are treated as a return of 0, heavily penalizing unstable policies.
- Imitation learning (Behavioral Cloning) sometimes outperforms complex offline RL algorithms due to easier tuning and robustness to noisy real-world observations.

## Evidence (verbatim from paper)

> Table 1 shows the average returns the teams achieved on all task/dataset combinations in the real-robot phase. The last column contains the overall score which is obtained by averaging these returns. (failed runs correspond to a return of 0).

## Citation

```bibtex
@misc{gurtler2023realrobotchallenge,
  title={Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World},
  author={Gurtler et al. (2023)},
  year={2023},
  note={arXiv:2308.07741}
}
```

- arXiv: 2308.07741