Mewl Eval

---
name: mewl-eval
description: This benchmark probes few-shot multimodal word learning under referential uncertainty by testing cross-situational reasoning, semantic bootstrapping, and pragmatic inference. It evaluates how well vision-language and language models generalize from limited examples to name attributes, objects, relations, numbers, and pragmatic concepts compared to human baselines. Use when the user wants to benchmark on MEWL, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.00503
  bibtex_key: jiang2023mewl
  confidence: high
---

# mewl-eval

> MEWL: Few-shot multimodal word learning with referential uncertainty — Jiang et al. (2023) (arXiv:2306.00503, 2023)

## What this evaluates

This benchmark probes few-shot multimodal word learning under referential uncertainty by testing cross-situational reasoning, semantic bootstrapping, and pragmatic inference. It evaluates how well vision-language and language models generalize from limited examples to name attributes, objects, relations, numbers, and pragmatic concepts compared to human baselines.

## Datasets

- **MEWL** — total ?; splits: train (-1), test (-1); repo https://github.com/jianggy/MEWL

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard classification accuracy calculated as the proportion of correct predictions out of total test instances. Reported as a percentage per task and averaged across all nine tasks.

## Input / output format

**Input**: Multimodal input consisting of one or more reference images and a text prompt/caption. The text prompt varies by task type: object-centric captions for attribute naming, relative spatial descriptions for relational tasks, and ground-truth pointing cues for pragmatic tasks.

**Output**: A single word or class label corresponding to the target concept (e.g., shape, color, material, object, composite, relation, bootstrap, number, or pragmatic term).

## Scoring recipe

```python
```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

# Per-task accuracy
per_task_scores = {task: compute_accuracy(preds[task], golds[task]) for task in tasks}
# Overall average
overall_avg = sum(per_task_scores.values()) / len(per_task_scores)
```
```

## Common pitfalls

- The benchmark simplifies the word learning challenge by injecting strong human bias via detailed captions and ground-truth pointing, which circumvents true concept abstraction and makes it more translation-like.
- GPT-3.5 is evaluated zero-shot via API (temperature 0), while all other models are fine-tuned on the training sets, creating an unfair baseline comparison.
- Human participants are evaluated on a randomly drawn 10-question subset from the test set per task, whereas models are evaluated on the full test set, complicating direct score comparison.

## Evidence (verbatim from paper)

> We report the model performance on the test sets. Table 2 summarizes the performance of both machines and humans, with result visualization in Figure 3. For each of the nine tasks, every participant was presented with a randomly drawn ten-question subset from the task's test set.

## Citation

```bibtex
@misc{jiang2023mewl,
  title={MEWL: Few-shot multimodal word learning with referential uncertainty},
  author={Jiang et al. (2023)},
  year={2023},
  note={arXiv:2306.00503}
}
```

- arXiv: 2306.00503