Ra Dt Icl Eval

---
name: ra-dt-icl-eval
description: This evaluation probes the in-context learning (ICL) capabilities of reinforcement learning agents across diverse environments, including grid-worlds, robotics simulators, and video games. It measures how effectively an agent can leverage retrieved past experiences to improve its policy over consecutive interaction trials without weight updates. Use when the user wants to benchmark on Dark-Room, Dark Key-Door, MazeRunner, Meta-World, DMControl, Procgen, or asks about evaluating this task. Reports mean reward.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.07071
  bibtex_key: schmied2024radt
  confidence: high
---

# ra-dt-icl-eval

> Retrieval-Augmented Decision Transformer: External Memory for In-context RL — Schmied et al. (2024) (arXiv:2410.07071, 2024)

## What this evaluates

This evaluation probes the in-context learning (ICL) capabilities of reinforcement learning agents across diverse environments, including grid-worlds, robotics simulators, and video games. It measures how effectively an agent can leverage retrieved past experiences to improve its policy over consecutive interaction trials without weight updates.

## Datasets

- **Dark-Room** — total ?; splits: train (80), test (20)
- **Dark Key-Door** — total ?; splits: train (80), test (20)
- **MazeRunner** — total ?; splits: train (100), test (20)
- **Meta-World** — total ?; splits: train (45), test (5)
- **DMControl** — total ?; splits: train (11), test (5)
- **Procgen** — total ?; splits: train (12), test (4)

## Metrics

- `mean reward` **(primary)** — range: other
  - Average cumulative reward obtained over a fixed number of ICL trials (episodes). Reported as the mean across tasks with 95% confidence intervals over 3 random seeds.

## Input / output format

**Input**: State observations (e.g., x-y coordinates, continuous Lidar-like depth vectors, or 64x64 RGB images) concatenated with retrieved sub-trajectories (state, action, reward sequences) or full context windows for baselines.

**Output**: Discrete action indices (for grid-worlds and Procgen) or continuous action vectors (for Meta-World and DMControl).

## Scoring recipe

```python
rewards = []
for trial in range(num_icl_trials):
    episode_reward = 0
    for step in range(max_steps):
        action = model.predict(state, context)
        next_state, reward, done, _ = env.step(action)
        episode_reward += reward
        state = next_state
        if done: break
    rewards.append(episode_reward)
mean_reward = np.mean(rewards)
```

## Common pitfalls

- Confusing standard RL evaluation (single long episode or fixed steps) with ICL evaluation, which requires measuring performance improvement across multiple consecutive trials where past episodes are stored in memory.
- Failing to report 95% confidence intervals over multiple random seeds, as the paper explicitly requires this for fair comparison across methods.
- Mixing up training and evaluation task splits, particularly in Procgen (PG12-Seen vs PG12-Unseen vs PG4) and Meta-World/DMControl, which have distinct holdout sets.

## Evidence (verbatim from paper)

> We evaluate each agent for 40 episodes on each of the 20 evaluation tasks and report mean reward (+ 95% CI, 3 seeds).

## Citation

```bibtex
@misc{schmied2024radt,
  title={Retrieval-Augmented Decision Transformer: External Memory for In-context RL},
  author={Schmied et al. (2024)},
  year={2024},
  note={arXiv:2410.07071}
}
```

- arXiv: 2410.07071