Chumor 1.0 Eval

---
name: chumor-1.0-eval
description: Evaluates large language models' ability to understand and explain culturally nuanced Chinese internet humor. It measures how well models can generate human-preferred, two-sentence explanations for jokes derived from the Chinese platform Ruo Zhi Ba. Use when the user wants to benchmark on Chumor 1.0, or asks about evaluating this task. Reports winning rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.12754
  bibtex_key: he2024chumor
  confidence: high
---

# chumor-1.0-eval

> Chumor 1.0: A Truly Funny and Challenging Chinese Humor Understanding Dataset from Ruo Zhi Ba — He et al. (2024) (arXiv:2406.12754, 2024)

## What this evaluates

Evaluates large language models' ability to understand and explain culturally nuanced Chinese internet humor. It measures how well models can generate human-preferred, two-sentence explanations for jokes derived from the Chinese platform Ruo Zhi Ba.

## Datasets

- **Chumor 1.0** — total 1951; splits: test (-1); repo https://github.com/dnaihao/Chumor-dataset

## Metrics

- `winning rate` **(primary)** — range: percent
  - The proportion of jokes where the LLM's two-sentence explanation is preferred over the human explanation based on majority vote from three native Chinese annotators. Cases where all annotators disagree are labeled 'Undecided' and excluded from the denominator.

## Input / output format

**Input**: A single joke text from the Chumor 1.0 dataset.

**Output**: A two-sentence Chinese explanation of the joke's humor.

## Scoring recipe

```python
llm_wins = 0
total_evaluated = 0
for joke in dataset:
    llm_exp = generate_explanation(joke)
    human_exp = gold_explanation[joke]
    votes = [annotate_preference(llm_exp, human_exp) for _ in range(3)]
    winner = majority_vote(votes)
    if winner == 'LLM':
        llm_wins += 1
    elif winner != 'Undecided':
        total_evaluated += 1
winning_rate = llm_wins / total_evaluated
```

## Common pitfalls

- The evaluation relies on human A/B preference testing rather than automated metrics, making it resource-intensive and subjective.
- Annotators are limited to six college students, which may not fully represent the broader Chinese-speaking population's humor perception.
- The 'Undecided' category for unanimous disagreement is excluded from the winning rate calculation, which can skew results if disagreement is high.

## Evidence (verbatim from paper)

> We employ the winning rate as our measure to compare LLMs’ explanation versus human explanation. We take the majority vote among all annotators for each example.

## Citation

```bibtex
@misc{he2024chumor,
  title={Chumor 1.0: A Truly Funny and Challenging Chinese Humor Understanding Dataset from Ruo Zhi Ba},
  author={He et al. (2024)},
  year={2024},
  note={arXiv:2406.12754}
}
```

- arXiv: 2406.12754