New Yorker Caption Humor Eval

---
name: new-yorker-caption-humor-eval
description: Evaluates AI's ability to understand humor through multimodal and text-based tasks, including matching captions to cartoons, ranking caption quality, and generating humorous explanations. It probes indirect allusion, cultural context, and visual-linguistic reasoning. Use when the user wants to benchmark on New Yorker Caption Contest, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.06293
  bibtex_key: hessel2022newyorkerhumor
  confidence: high
---

# new-yorker-caption-humor-eval

> Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest — Hessel et al. (2022) (arXiv:2209.06293, 2022)

## What this evaluates

Evaluates AI's ability to understand humor through multimodal and text-based tasks, including matching captions to cartoons, ranking caption quality, and generating humorous explanations. It probes indirect allusion, cultural context, and visual-linguistic reasoning.

## Datasets

- **New Yorker Caption Contest** — total 704; splits: 5-fold CV (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly matched captions or selected winners against ground truth labels. Calculated as the mean of correct predictions over total instances.
- `CrowdAcc` — range: [0, 1]
  - Accuracy against crowd-sourced quality rankings for cartoon captions. Measures how well models predict general audience preferences.
- `NYAcc` — range: [0, 1]
  - Accuracy against New Yorker editor selections for cartoon captions. Measures alignment with professional editorial taste.
- `Pairwise Win Rate` — range: [0, 1]
  - Percentage of instances where one model's explanation is preferred over another's in a head-to-head comparison. Determined by majority vote of three crowd workers per instance.

## Input / output format

**Input**: For matching/ranking: cartoon image (From Pixels) or human-authored cartoon description (From Description) concatenated with multiple-choice caption options. For explanation: cartoon description and/or caption conditioned for autoregressive generation.

**Output**: For matching/ranking: a single letter corresponding to the selected caption choice. For explanation: a generated natural language text explaining the humor.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    return sum(p == g for p, g in zip(predictions, gold)) / len(gold)

def compute_pairwise_win_rate(model_a_exps, model_b_exps, gold_votes):
    wins_a = 0
    for a, b, votes in zip(model_a_exps, model_b_exps, gold_votes):
        if votes.count('A') > votes.count('B'):
            wins_a += 1
    return wins_a / len(gold_votes)
```

## Common pitfalls

- Human performance is not an upper bound for matching/ranking because labels reflect varying tastes and are not from a single annotator.
- Surface-level automatic metrics like BLEU-4 and Rouge-L do not correlate with human preference for explanation quality.
- Zero-shot prompting universally incurs performance drops compared to few-shot or fine-tuned setups across all tasks.

## Evidence (verbatim from paper)

> Table 2 contains the results. Among the from description models, GPT-4 (5-shot) generally performs best, e.g., achieving 84.5% accuracy on matching. It (and fine-tuned GPT-3) also perform better at predicting New Yorker editor selections than our three humans (column NYAcc: GPT-3 69.8 vs. Human estimate, 64.6), but underperform at predicting crowd selections (CrowdAcc column: GPT-4 73.3 vs. 83.7).

## Citation

```bibtex
@misc{hessel2022newyorkerhumor,
  title={Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest},
  author={Hessel et al. (2022)},
  year={2022},
  note={arXiv:2209.06293}
}
```

- arXiv: 2209.06293