Culemo Eval

---
name: culemo-eval
description: Probes LLMs' cross-cultural emotion understanding by testing their ability to predict emotions and sentiments across six languages, specifically examining how prompt language and explicit country context influence model performance. Use when the user wants to benchmark on CULEMO, or asks about evaluating this task. Reports emotion prediction.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.10688
  bibtex_key: belay2025culemo
  confidence: medium
---

# culemo-eval

> CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding — Belay et al. (2025) (arXiv:2503.10688, 2025)

## What this evaluates

Probes LLMs' cross-cultural emotion understanding by testing their ability to predict emotions and sentiments across six languages, specifically examining how prompt language and explicit country context influence model performance.

## Datasets

- **CULEMO** — total ?; splits: test (-1); repo https://github.com/llm-for-emotion/culemo

## Metrics

- `emotion prediction` **(primary)** — range: [0, 1]
  - Accuracy of predicted emotion categories compared to gold annotations, evaluated via zero-shot text generation.
- `sentiment analysis` — range: [0, 1]
  - Accuracy of predicted sentiment polarity compared to gold annotations, evaluated via zero-shot text generation.

## Input / output format

**Input**: Instruction, input text, and expected answer format in either English or one of five target languages (Arabic, Amharic, German, Hindi, Spanish), optionally prefixed with explicit country context (e.g., 'You live in <<country name>>,').

**Output**: Generated text containing the predicted emotion or sentiment label, extracted using the PEDANTS tool.

## Scoring recipe

```python
def compute_metric(predictions, gold):
    # Extract predicted label from model output using PEDANTS tool
    pred_labels = [extract_label(pred) for pred in predictions]
    # Calculate accuracy
    correct = sum(1 for p, g in zip(pred_labels, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Using English prompts for non-English cultures significantly underperforms compared to in-language prompting with country context.
- Relying on translated datasets introduces cultural bias; native cultural annotations are required for valid evaluation.
- Zero-shot text generation requires robust label extraction; without tools like PEDANTS, parsing free-form model outputs is error-prone.

## Evidence (verbatim from paper)

> Specifically, we explore culture-aware emotion understanding via two main tasks: (1) emotion prediction and (2) sentiment analysis. All tested models are instruction-fine-tuned, except for the Aya-expanse model. We also experiment with prompts that do and do not include explicit country context, using the phrase "You live in <<country name>>," (where <<country name>> is one of the six targeted countries: UAE, USA, Ethiopia, Germany, India, and Mexico). Each task is framed as a text-generation problem, and the models are evaluated in a zero-shot setting.

## Citation

```bibtex
@misc{belay2025culemo,
  title={CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding},
  author={Belay et al. (2025)},
  year={2025},
  note={arXiv:2503.10688}
}
```

- arXiv: 2503.10688