Cult Eval Eval

---
name: cult-eval-eval
description: This benchmark probes a machine translation model's ability to accurately translate culture-loaded expressions (idioms, proverbs, culture-specific items) while preserving their figurative, contextual, and cultural meanings. It evaluates whether models can avoid literal or superficial translations that strip away culturally grounded nuances. Use when the user wants to benchmark on CulT-Eval, or asks about evaluating this task. Reports ACRE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.17303
  bibtex_key: han2026culteval
  confidence: high
---

# cult-eval-eval

> From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation — Han et al. (2026) (arXiv:2603.17303, 2026)

## What this evaluates

This benchmark probes a machine translation model's ability to accurately translate culture-loaded expressions (idioms, proverbs, culture-specific items) while preserving their figurative, contextual, and cultural meanings. It evaluates whether models can avoid literal or superficial translations that strip away culturally grounded nuances.

## Datasets

- **CulT-Eval** — total 7959; splits: test (-1)

## Metrics

- `ACRE` **(primary)** — range: [0, 1]
  - A taxonomy-aware evaluation metric designed to systematically identify and quantify culturally induced meaning deviations. It aligns model outputs with human annotations across a five-way cultural taxonomy (Material, Social, Linguistic, Religious, Ecological) to capture nuances beyond surface-level similarity.
- `Pearson r` — range: [-1, 1]
  - Standard Pearson correlation coefficient measuring the linear relationship between model metric scores and human annotation scores.
- `Spearman ρ` — range: [-1, 1]
  - Standard Spearman rank correlation coefficient measuring the monotonic relationship between model metric scores and human annotation scores.

## Input / output format

**Input**: Chinese source sentence containing a culture-loaded expression, paired with a verified English reference translation.

**Output**: Model-generated English translation of the Chinese source sentence.

## Scoring recipe

```python
def evaluate(predictions, references, human_scores):
    # Compute taxonomy-aware scores for each prediction
    model_scores = [compute_acre(pred, ref) for pred, ref in zip(predictions, references)]
    # Align with human judgment scores
    pearson_r = pearsonr(model_scores, human_scores)
    spearman_rho = spearmanr(model_scores, human_scores)
    return {
        'ACRE': mean(model_scores),
        'Pearson_r': pearson_r,
        'Spearman_rho': spearman_rho
    }
```

## Common pitfalls

- Standard surface-level metrics like BLEU and COMET often yield high scores but fail to detect critical failures in preserving figurative or context-dependent cultural meaning.
- Evaluation requires span-level inspection; models must correctly translate the specific culture-loaded expression span rather than just achieving sentence-level fluency.
- Instances with weak contextual support or insufficient cultural salience are explicitly filtered out, so results may not generalize to ambiguous or low-context cultural terms.

## Evidence (verbatim from paper)

> Table 2. Pearson (r) and Spearman (ρ) correlation coefficients between metrics and human annotations on CulT-Eval.
| Metric | Pearson $r$ | Spearman $
ho$ |
| --- | --- | --- |
| BLEU | 30.2 | 28.4 |
| ChrF++ | 22.4 | 20.1 |
| BERTScore | 27.5 | 25.3 |
| COMET | 44.5 | 39.0 |
| MetricX-QE | 24.6 | 22.8 |
| ACRE (Ours) | 68.4 | 65.1 |

## Citation

```bibtex
@misc{han2026culteval,
  title={From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation},
  author={Han et al. (2026)},
  year={2026},
  note={arXiv:2603.17303}
}
```

- arXiv: 2603.17303