Histoires Morales Eval

---
name: histoires-morales-eval
description: Evaluates the linguistic quality and cultural appropriateness of the Histoires Morales dataset. It measures reference-free translation accuracy and assesses whether moral norms and actions align with native French speakers' cultural values. Use when the user wants to benchmark on Histoires Morales, or asks about evaluating this task. Reports CometKiwi22.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.17117
  bibtex_key: leteno2025histoiresmorales
  confidence: high
---

# histoires-morales-eval

> Histoires Morales: A French Dataset for Assessing Moral Alignment — Leteno et al. (2025) (arXiv:2501.17117, 2025)

## What this evaluates

Evaluates the linguistic quality and cultural appropriateness of the Histoires Morales dataset. It measures reference-free translation accuracy and assesses whether moral norms and actions align with native French speakers' cultural values.

## Datasets

- **Histoires Morales** — total 12000; splits: full (12000); repo https://github.com/upunaprosk/histoires-morales

## Metrics

- `CometKiwi22` **(primary)** — range: [0, 1]
  - Reference-free quality estimation metric for sentence-level translation. Outputs a continuous score between 0 and 1, where higher values indicate better translation quality.
- `cultural_alignment_agreement` — range: percent
  - Categorical label derived from 4 native French annotators. Agreement if fewer than 2 annotators disagree, Disagreement if more than 2 disagree, Uncertainty if exactly 2 disagree.

## Input / output format

**Input**: Translated French sentences categorized into Norm, Situation, Intention, Moral action, Moral consequence, Immoral action, or Immoral consequence.

**Output**: For translation: a float score between 0 and 1. For cultural alignment: a categorical label (Agreement, Disagreement, or Uncertainty).

## Scoring recipe

```python
def compute_cometkiwi22(predictions, gold):
    return cometkiwi22_model.predict(predictions, gold)  # Returns float in [0, 1]

def compute_cultural_agreement(predictions, gold):
    disagree_count = sum(1 for p in predictions if p == 'disagree')
    if disagree_count < 2:
        return 'Agreement'
    elif disagree_count > 2:
        return 'Disagreement'
    else:
        return 'Uncertainty'
```

## Common pitfalls

- Reference-free QE models like CometKiwi22 may penalize context-sensitive phrasal verbs or collocations that are actually correct and culturally appropriate translations.
- Cultural alignment assessment relies on a small sample (500 items) and subjective annotator thresholds, which may not generalize to broader French-speaking populations or nuanced moral situations.

## Evidence (verbatim from paper)

> We measure the quality of translation with the CometKiwi22 reference-free quality estimation (QE) metric introduced by Rei et al., [2022]. This metric is suitable for sentence and word-level QE and supports English-to-French translations, with values between 0 and 1, and higher values indicating better translations.

## Citation

```bibtex
@misc{leteno2025histoiresmorales,
  title={Histoires Morales: A French Dataset for Assessing Moral Alignment},
  author={Leteno et al. (2025)},
  year={2025},
  note={arXiv:2501.17117}
}
```

- arXiv: 2501.17117