Only Connect Wall Eval

---
name: only-connect-wall-eval
description: Evaluates creative problem-solving and associative reasoning by testing whether models can correctly group words and identify connections, specifically probing susceptibility to cognitive fixation effects when misleading red herring clues are present. Use when the user wants to benchmark on Only Connect Wall (OCW), or asks about evaluating this task. Reports grouping_evaluation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.11167
  bibtex_key: alavinaeini2023onlyconnectwall
  confidence: medium
---

# only-connect-wall-eval

> Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset — Alavi Naeini et al. (2023) (arXiv:2306.11167, 2023)

## What this evaluates

Evaluates creative problem-solving and associative reasoning by testing whether models can correctly group words and identify connections, specifically probing susceptibility to cognitive fixation effects when misleading red herring clues are present.

## Datasets

- **Only Connect Wall (OCW)** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TaatiTeam/OCW

## Metrics

- `grouping_evaluation` **(primary)** — range: [0, 1]
  - Exact match accuracy of the four predicted word groups to the four ground-truth groups per wall. Evaluated across 16 runs with different seeds and randomized word orderings to account for initialization variance.

## Input / output format

**Input**: Task 1: A list of 16 clue words per wall. Task 2: A list of 16 clue words (or a solved wall without connections) provided in a few-shot prompt with in-context examples.

**Output**: Task 1: Four predicted groups of words. Task 2: Four predicted connections. Outputs are post-processed to contain exactly 4 predictions, padded with empty strings if fewer are generated.

## Scoring recipe

```python
def score_grouping(predictions, gold):
    # predictions and gold are lists of 4 lists of words
    best_match = 0
    for perm in permutations(range(4)):
        if all(set(predictions[i]) == set(gold[perm[i]]) for i in range(4)):
            best_match = 1
            break
    return best_match
```

## Common pitfalls

- Contextual embeddings are sensitive to word order; the paper randomizes word order across 16 runs to mitigate this.
- Out-of-vocabulary clues are handled via mean pooling or BPEmb sub-words, which can affect embedding quality.
- LLM outputs must be strictly padded to 4 groups/connections to match the evaluation format.

## Evidence (verbatim from paper)

> For the grouping task evaluation (§2.2), we use clustering algorithms on word-embeddings of the sixteen clue words in each wall, to group them into four predicted groups that are subsequently evaluated against the four ground-truth groups for each wall. ... We developed our prompts on the validation set and reported the final performance on the test set.

## Citation

```bibtex
@misc{alavinaeini2023onlyconnectwall,
  title={Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset},
  author={Alavi Naeini et al. (2023)},
  year={2023},
  note={arXiv:2306.11167}
}
```

- arXiv: 2306.11167