Much Eval

---
name: much-eval
description: Evaluates the ability of logit-based uncertainty quantification (UQ) methods to predict claim-level hallucination (factuality) in multilingual LLM outputs. It measures how well token-level confidence scores, when aggregated, correlate with ground-truth factuality labels across different languages and model configurations. Use when the user wants to benchmark on MUCH, or asks about evaluating this task. Reports ROC-AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.17081
  bibtex_key: dentan2025much
  confidence: high
---

# much-eval

> MUCH: A Multilingual Claim Hallucination Benchmark — Dentan et al. (2025) (arXiv:2511.17081, 2025)

## What this evaluates

Evaluates the ability of logit-based uncertainty quantification (UQ) methods to predict claim-level hallucination (factuality) in multilingual LLM outputs. It measures how well token-level confidence scores, when aggregated, correlate with ground-truth factuality labels across different languages and model configurations.

## Datasets

- **MUCH** — total 20751; splits: test (-1); repo https://github.com/orailix/much

## Metrics

- `ROC-AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
- `PR-AUC` — range: [0, 1]
  - Area under the Precision-Recall curve, measuring the trade-off between precision and recall across all thresholds, particularly sensitive to class imbalance.

## Input / output format

**Input**: Text segments (claims) produced by a deterministic segmenter, along with per-token logits from the LLM and ground-truth factuality labels.

**Output**: Claim-level uncertainty score (continuous) derived by aggregating token-level UQ scores using a specified aggregator (mean, max, geometric mean, or product).

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # predictions: list of aggregated claim-level UQ scores
    # gold: list of binary factuality labels (1=hallucinated, 0=factual)
    roc_auc = roc_auc_score(gold, predictions)
    pr_auc = average_precision_score(gold, predictions)
    return {"ROC-AUC": roc_auc, "PR-AUC": pr_auc}
```

## Common pitfalls

- Aggregator choice drastically changes results; the product aggregator consistently outperforms mean/max/geometric mean.
- Baselines like SAR and CCP rely on NLI models optimized only for English, causing severe performance drops in Spanish and other languages.
- UQ computation time can exceed generation time (e.g., CCP takes ~197% of generation time), making real-time monitoring infeasible without optimization.

## Evidence (verbatim from paper)

> Following*(Fadeeva et al., [2024])*, we evaluated four aggregators: the (arithmetic) mean of token values in the claim, the maximum, the geometric mean, and the product. Consistent with the observations of*(Fadeeva et al., [2024])*, our best results are obtained when using the product as aggregator. We report ROC-AUC and Precision–Recall (PR) AUC.

## Citation

```bibtex
@misc{dentan2025much,
  title={MUCH: A Multilingual Claim Hallucination Benchmark},
  author={Dentan et al. (2025)},
  year={2025},
  note={arXiv:2511.17081}
}
```

- arXiv: 2511.17081