Halma Eval

---
name: halma-eval
description: Evaluates an agent's ability to learn humanlike abstractions and affordances for rapid problem solving in a structured visual game. It probes three levels of generalization: perceptual recognition, conceptual abstraction of semantics, and algorithmic strategy formation under limited training exposure. Use when the user wants to benchmark on HALMA, or asks about evaluating this task. Reports goal_reaching ($
ho_g$).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.11344
  bibtex_key: xie2021halma
  confidence: high
---

# halma-eval

> HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving — Xie et al. (2021) (arXiv:2102.11344, 2021)

## What this evaluates

Evaluates an agent's ability to learn humanlike abstractions and affordances for rapid problem solving in a structured visual game. It probes three levels of generalization: perceptual recognition, conceptual abstraction of semantics, and algorithmic strategy formation under limited training exposure.

## Datasets

- **HALMA** — total ?; splits: train (-1), random_split (-1), ST_generalization (-1), AfT_generalization (-1), AnT_generalization (-1)

## Metrics

- `valid_move_ratio ($
ho_a$)` — range: percent
  - The proportion of executed moves that are legally valid within the game rules.
- `goal_reaching ($
ho_g$)` **(primary)** — range: percent
  - The percentage of test episodes in which the agent successfully reaches the designated goal state.
- `efficiency ($
ho_p$)` — range: percent
  - A measure of the agent's path or action efficiency relative to an optimal or maximum allowed length.

## Input / output format

**Input**: Symbolic one-hot observations representing ground-truth game state, or raw visual observation panels.

**Output**: Discrete actions/moves within the Super Halma environment.

## Scoring recipe

```python
def score(predictions, gold):
    valid_ratio = sum(1 for a in predictions if is_valid(a)) / len(predictions)
    goal_reached = sum(1 for ep in predictions if ep.reaches_goal()) / len(predictions)
    efficiency = compute_efficiency(predictions) # e.g., optimal_steps / actual_steps
    return valid_ratio, goal_reached, efficiency
```

## Common pitfalls

- Assuming a high goal-reaching rate implies the agent understands affordances or temporal grammar; low valid move ratios reveal brittle rule comprehension.
- Confusing random split evaluation with true out-of-distribution generalization tests (ST, AfT, AnT) that require compositional abstraction.
- Using standard CNNs without disentanglement priors causes latent space failures that misrepresent the agent's ability to recognize objects and colors.

## Evidence (verbatim from paper)

> All agents show relatively low valid move ratio $m{
ho_{a}}$ in tests of random split, indicating their understanding of affordance is brittle even with the ground-truth semantics. Under this precondition, we find that all agents can still perform relatively well in terms of goal-reaching $m{
ho_{g}}$ and efficiency $m{
ho_{p}}$ in random splits.

## Citation

```bibtex
@misc{xie2021halma,
  title={HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving},
  author={Xie et al. (2021)},
  year={2021},
  note={arXiv:2102.11344}
}
```

- arXiv: 2102.11344