Handmethat Eval

---
name: handmethat-eval
description: Evaluates a robot's ability to understand ambiguous human instructions and infer human subgoals in physically and socially complex household environments. It probes pragmatic reasoning, goal recognition, and collaborative task completion under partial observability. Use when the user wants to benchmark on HandMeThat, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.03779
  bibtex_key: wan2023handmethat
  confidence: high
---

# handmethat-eval

> HandMeThat: Human-Robot Communication in Physical and Social Environments — Wan et al. (2023) (arXiv:2310.03779, 2023)

## What this evaluates

Evaluates a robot's ability to understand ambiguous human instructions and infer human subgoals in physically and socially complex household environments. It probes pragmatic reasoning, goal recognition, and collaborative task completion under partial observability.

## Datasets

- **HandMeThat** — total 10000; splits: test (1000)

## Metrics

- `average score` — range: other
  - Mean reward per episode, averaged over 1,000 test episodes.
- `success rate` **(primary)** — range: percent
  - Percentage of episodes where the agent achieves the goal within a maximum of 40 steps.
- `average moves (successful)` — range: other
  - Mean number of steps taken to complete the task, averaged only over successful episodes.

## Input / output format

**Input**: Textual instruction/utterance (potentially ambiguous), task description, and environment observation (object states and locations) provided in either fully or partially observable settings.

**Output**: Discrete action selection from a valid action space (15–30 options per step), typically involving picking/placing objects or asking clarification questions.

## Scoring recipe

```python
def evaluate(predictions, gold, max_steps=40):
    successful_moves = []
    total_reward = 0.0
    for pred_seq, episode in zip(predictions, gold):
        total_reward += episode_reward(episode)
        if len(pred_seq) <= max_steps and check_goal_reached(pred_seq, episode):
            successful_moves.append(len(pred_seq))
    success_rate = (len(successful_moves) / len(gold)) * 100
    avg_moves = sum(successful_moves) / len(successful_moves) if successful_moves else 0
    avg_score = total_reward / len(gold)
    return avg_score, success_rate, avg_moves
```

## Common pitfalls

- The Heuristic baseline assumes full observability and ground-truth symbolic utterance representations, making direct comparison with learning models in partial observability settings unfair.
- Level 3 and 4 tasks are intentionally designed to be 'intrinsically unsolvable' due to insufficient contextual information, which can be mistaken for model failure rather than a benchmark feature requiring clarification.
- DRRN baselines fail primarily due to a large action space (15-30 valid actions) and sparse rewards, not necessarily a lack of pragmatic reasoning capability.

## Evidence (verbatim from paper)

> We consider three evaluation metrics: 1) the average score of the model; 2) the success rate that the model achieves the goal within limited steps (40); 3) the average number of moves of successful episodes. Scores are averaged on 1,000 episodes.

## Citation

```bibtex
@misc{wan2023handmethat,
  title={HandMeThat: Human-Robot Communication in Physical and Social Environments},
  author={Wan et al. (2023)},
  year={2023},
  note={arXiv:2310.03779}
}
```

- arXiv: 2310.03779