Unclearinstruct Eval

---
name: unclearinstruct-eval
description: Evaluates a robot's ability to infer human goals and execute household tasks from noisy, accented, or mispronounced spoken instructions. It probes robust speech perception, joint planning, and Theory of Mind in embodied human-robot collaboration under mixed-observability conditions. Use when the user wants to benchmark on UnclearInstruct, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.10849
  bibtex_key: ying2024pragmatic
  confidence: high
---

# unclearinstruct-eval

> Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind — Ying et al. (2024) (arXiv:2409.10849, 2024)

## What this evaluates

Evaluates a robot's ability to infer human goals and execute household tasks from noisy, accented, or mispronounced spoken instructions. It probes robust speech perception, joint planning, and Theory of Mind in embodied human-robot collaboration under mixed-observability conditions.

## Datasets

- **UnclearInstruct** — total 3080; splits: test (3080)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Percentage of trials where the robot correctly infers the intended robot goal from the speech instruction.
- `Speedup` — range: [0, 1]
  - Ratio of single-agent task completion timesteps to team completion timesteps, minus one: L_single / L_team - 1.
- `Word Error Rate (WER)` — range: [0, 1]
  - Percentage of word errors in the generated transcript relative to the total words spoken. A perfect transcription yields 0.

## Input / output format

**Input**: Noisy speech audio file (original, noise-injected, accented, or mispronounced), optionally accompanied by text descriptions of visual observations and the goal space.

**Output**: A symbolic robot goal specification (e.g., item types and quantities to pick up) and/or a text transcription of the speech instruction.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p['goal'] == g['goal'])
    accuracy = correct / len(gold)
    speedups = [(g['timesteps_single'] / p['timesteps']) - 1 for p, g in zip(predictions, gold)]
    avg_speedup = sum(speedups) / len(speedups)
    wer_sum = sum(edit_distance(p['transcript'], g['transcript']) / len(g['transcript']) for p, g in zip(predictions, gold))
    wer = wer_sum / len(gold)
    return {'accuracy': accuracy, 'speedup': avg_speedup, 'wer': wer}
```

## Common pitfalls

- The speech conditions (noise, accents, mispronunciations) are synthetically generated using external tools rather than naturally recorded, limiting real-world acoustic generalization.
- Speedup is tightly coupled to the VirtualHome simulator's timestep definition and baseline comparison (single human vs. human-robot team), making cross-platform comparison difficult.
- WER for SIFToM is computed on Whisper transcripts rather than end-to-end model outputs, which may inflate comparability with pure ASR baselines.

## Evidence (verbatim from paper)

> We evaluated the model performance with three metrics: accuracy rate, speedup, and word error rare (WER). a) Accuracy: We measured inference accuracy by the percentage of trials that the robot inferred the correct robot goal from speech instructions. b) Speedup: Following past work on human-robot collaboration in VirtualHome [35], we computed the speedup of the baseline models against a single-agent (human alone) baseline. The speedup is computed as $L_{\text{single}} / L_{\text{team}} - 1$, where $L_{\text{single}}$ is the timesteps it takes for the human agent to complete the task alone without robot assistance and $L_{\text{team}}$ is the timesteps for the human-robot team. c) Word Error Rate: We followed existing work on ASR, which calculated WER as the percentage of errors in a transcript compared to the total words spoken. A perfect transcription has a WER of 0.

## Citation

```bibtex
@misc{ying2024pragmatic,
  title={Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind},
  author={Ying et al. (2024)},
  year={2024},
  note={arXiv:2409.10849}
}
```

- arXiv: 2409.10849