Gscan Eval

---
name: gscan-eval
description: Evaluates systematic generalization in grounded language understanding by testing whether models can interpret natural language commands within dynamic grid-world environments. It probes compositional generalization across novel object properties, navigation directions, contextual size references, action-argument bindings, and adverbial modifiers. Use when the user wants to benchmark on gSCAN, or asks about evaluating this task. Reports exact match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.05161
  bibtex_key: ruis2020gscan
  confidence: high
---

# gscan-eval

> A Benchmark for Systematic Generalization in Grounded Language Understanding — Ruis et al. (2020) (arXiv:2003.05161, 2020)

## What this evaluates

Evaluates systematic generalization in grounded language understanding by testing whether models can interpret natural language commands within dynamic grid-world environments. It probes compositional generalization across novel object properties, navigation directions, contextual size references, action-argument bindings, and adverbial modifiers.

## Datasets

- **gSCAN** — total ?; splits: random (19282), novel-object-properties-red (-1), novel-object-properties-yellow (-1), novel-direction (-1), novel-contextual-references (-1), novel-action-arguments (-1), novel-adverbs-fewshot (-1), novel-adverbs-combination (-1); repo https://github.com/LauraRuis/groundedSCAN

## Metrics

- `exact match accuracy` **(primary)** — range: percent
  - Calculated as the percentage of test examples where the model's predicted action sequence exactly matches the gold action sequence. Partial matches or step-wise correctness do not receive credit.

## Input / output format

**Input**: A natural language instruction (e.g., 'walk to the small red circle') paired with a dynamic grid-world visual state containing objects of varying colors, shapes, and sizes at specific coordinates.

**Output**: A sequence of discrete navigation/interaction actions (e.g., walk, L_turn, R_turn, stay, push, pull) that moves the agent from its starting position to the target object and performs the required interaction.

## Scoring recipe

```python
def compute_exact_match(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Exact match requires the entire action sequence to be identical; step-wise accuracy or partial path completion is not counted as correct.
- Splits are grounded in specific world-state configurations (e.g., relative size, object weight, spatial direction), not just linguistic variations, so models cannot rely on purely syntactic generalization.
- The benchmark uses synthetic grid-world environments, so performance does not directly transfer to real-world robotics or complex visual grounding tasks without adaptation.

## Evidence (verbatim from paper)

> The baseline model achieves near perfect exact match accuracy ($97.69\%\pm 0.22$ mean over 3 runs, reported with standard deviation) on the 19,282 test examples, where exact match means that the entire action sequence is produced correctly.

## Citation

```bibtex
@misc{ruis2020gscan,
  title={A Benchmark for Systematic Generalization in Grounded Language Understanding},
  author={Ruis et al. (2020)},
  year={2020},
  note={arXiv:2003.05161}
}
```

- arXiv: 2003.05161