MENLI

---
name: MENLI
description: Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions. Use when the user has predictions and gold and needs to compute Pearson correlation.
metadata:
  skill_kind: metric
  source_arxiv: 2208.07316
  bibtex_key: chen2022menli
  confidence: high
---

# MENLI

> MENLI: Robust Evaluation Metrics from Natural Language Inference — Chen et al. (2022) (arXiv:2208.07316, 2022)

## What this evaluates

Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions.

## Datasets

- **WMT15-17 / WMT20-21** — total ?; splits: test (-1)
- **RealSum** — total ?; splits: test (-1)
- **SummEval** — total ?; splits: test (-1)
- **PAWS / XPAWS / SEadv / Rank19** — total ?; splits: adversarial (-1)

## Metrics

- `Pearson correlation` **(primary)** — range: [-1, 1]
  - Measures the linear correlation between metric scores and human judgment scores (e.g., DA, MQM, or LitePyramids ratings) at segment or system level.
- `Kendall correlation` — range: [-1, 1]
  - Measures ordinal association between metric scores and human judgments on coherence, consistency, fluency, and relevance criteria.
- `Accuracy` — range: [0, 1]
  - Relative frequency that a metric correctly ranks the preferred candidate higher than the incorrect candidate in adversarial pairwise comparisons.

## Input / output format

**Input**: Source text (or reference) and candidate translation/summary. For adversarial tasks, pairs of candidates (one correct, one incorrect) are provided.

**Output**: A continuous scalar score indicating quality/relevance, or a binary preference decision for adversarial pairs.

## Scoring recipe

```python
def compute_menli(premise, hypothesis, nli_model, formula, direction):
    e, c, n = nli_model.predict(premise, hypothesis)
    score = apply_formula(e, c, n, formula)
    if direction == 'bidirectional':
        score = (score_fwd + score_bwd) / 2
    return score

def evaluate_against_human(menli_scores, human_ratings):
    return pearsonr(menli_scores, human_ratings)

def adversarial_accuracy(correct_scores, incorrect_scores):
    return mean([c > i for c, i in zip(correct_scores, incorrect_scores)])
```

## Common pitfalls

- Directionality matters: ref/src → cand vs ref/src ← cand vs bidirectional averaging yield different rankings; ref-free summarization only uses src → cand.
- Multi-reference aggregation: using max vs mean aggregation for reference summaries significantly changes correlation values on SummEval.
- NLI probability normalization: e, c, n must sum to 1; different fine-tuning datasets (SNLI, MNLI, ANLI, XNLI) drastically affect robustness.

## Evidence (verbatim from paper)

> In general, we check the metrics' preference between the two candidates and calculate accuracy: the relative frequency that the metrics correctly choose among the two alternatives. On MT standard benchmarks, we evaluate the metrics on both segment-level (where we correlate metrics scores to human judgements for individual sentences/segments in the datasets) and system-level (where we correlate the average metric scores to the average human scores over the segments generated by each system), using Pearson correlation as the performance indicator.

## Citation

```bibtex
@misc{chen2022menli,
  title={MENLI: Robust Evaluation Metrics from Natural Language Inference},
  author={Chen et al. (2022)},
  year={2022},
  note={arXiv:2208.07316}
}
```

- arXiv: 2208.07316