Lm Loss And Benchmark Eval

---
name: lm-loss-and-benchmark-eval
description: Evaluates the impact of multilingual data mixtures on language modeling capability and downstream task performance across multiple languages. It probes whether English dominance or high language count negatively interferes with multilingual model training. Use when the user wants to benchmark on mC4, FineWeb2, or asks about evaluating this task. Reports language modeling loss.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.25947
  bibtex_key: foroutan2025revisiting
  confidence: medium
---

# lm-loss-and-benchmark-eval

> Revisiting Multilingual Data Mixtures in Language Model Pretraining — Foroutan et al. (2025) (arXiv:2510.25947, 2025)

## What this evaluates

Evaluates the impact of multilingual data mixtures on language modeling capability and downstream task performance across multiple languages. It probes whether English dominance or high language count negatively interferes with multilingual model training.

## Datasets

- **mC4** — total ?; splits: validation (-1); HF `allenai/c4`
- **FineWeb2** — total ?; splits: validation (-1)

## Metrics

- `language modeling loss` **(primary)** — range: other
  - Standard token-level cross-entropy loss computed over a held-out validation set distinct from pretraining data. Lower values indicate better language modeling capability.
- `downstream task performance` — range: percent
  - Aggregated scores from a suite of multilingual benchmarks, computed per language and then averaged across non-English languages.

## Input / output format

**Input**: Tokenized text sequences from the held-out validation set or downstream benchmark tasks, processed through the Mistral-Nemo-Base-2407 tokenizer.

**Output**: Predicted probability distribution over the 131,000-token vocabulary for each token position; or task-specific predictions for downstream benchmarks.

## Scoring recipe

```python
def compute_lm_loss(predictions, targets):
    loss = 0.0
    for pred, target in zip(predictions, targets):
        loss += cross_entropy(pred, target)
    return loss / len(targets)

def aggregate_benchmark_scores(per_language_scores):
    non_english_scores = [s for lang, s in per_language_scores if lang != 'English']
    return sum(non_english_scores) / len(non_english_scores)
```

## Common pitfalls

- The exact composition of the 'suite of multilingual benchmarks' and their specific tasks are not detailed in the main text (referenced in Appendix B).
- The weighting scheme for the 'weighted average LM loss of non-English languages' and benchmark aggregation is not specified in the provided section.
- Random baselines are mentioned in figures but their exact calculation method (e.g., uniform vs. empirical) is not defined.

## Evidence (verbatim from paper)

> We evaluate our models by measuring their language modeling loss on a held-out validation set that is distinct from the pretraining data. In addition, we perform downstream task evaluations using a suite of multilingual benchmarks. For each model, we aggregate results by language to obtain a comprehensive score for every model-language pair.

## Citation

```bibtex
@misc{foroutan2025revisiting,
  title={Revisiting Multilingual Data Mixtures in Language Model Pretraining},
  author={Foroutan et al. (2025)},
  year={2025},
  note={arXiv:2510.25947}
}
```

- arXiv: 2510.25947