Multilingual Llm Downstream Eval

---
name: multilingual-llm-downstream-eval
description: Evaluates the downstream capabilities of multilingual LLMs trained on filtered pretraining data. It probes reading comprehension, general knowledge, natural language understanding, common-sense reasoning, and generative tasks across multiple languages. Use when the user wants to benchmark on FineTasks, SmolLM tasks suite, or asks about evaluating this task. Reports average rank.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.10361
  bibtex_key: messmer2025enhancing
  confidence: high
---

# multilingual-llm-downstream-eval

> Enhancing Multilingual LLM Pretraining with Model-Based Data Selection — Messmer et al. (2025) (arXiv:2502.10361, 2025)

## What this evaluates

Evaluates the downstream capabilities of multilingual LLMs trained on filtered pretraining data. It probes reading comprehension, general knowledge, natural language understanding, common-sense reasoning, and generative tasks across multiple languages.

## Datasets

- **FineTasks** — total ?; splits: test (-1)
- **SmolLM tasks suite** — total ?; splits: test (-1)

## Metrics

- `average rank` **(primary)** — range: other
  - Models are ranked per task based on accuracy (1 = best). The average rank is computed across all tasks and languages. Lower values indicate better overall performance.
- `task accuracy` — range: [0, 1]
  - Standard exact-match or option-selection accuracy per downstream benchmark task (e.g., MMLU, ARC, HellaSwag, Belebele).

## Input / output format

**Input**: Text prompts from downstream evaluation benchmarks (FineTasks/SmolLM) covering reading comprehension, general knowledge, NLU, common-sense reasoning, and generation.

**Output**: Model-generated text or selected options for each benchmark task.

## Scoring recipe

```python
def compute_average_rank(task_accuracies):
    # task_accuracies: dict of {task_name: {model_id: accuracy}}
    ranks_per_task = {}
    for task, scores in task_accuracies.items():
        sorted_models = sorted(scores.items(), key=lambda x: x[1], reverse=True)
        ranks_per_task[task] = {m: r + 1 for r, (m, _) in enumerate(sorted_models)}
    avg_ranks = {m: sum(ranks_per_task[t][m] for t in ranks_per_task) / len(ranks_per_task) 
                 for m in ranks_per_task[list(ranks_per_task.keys())[0]]}
    return avg_ranks
```

## Common pitfalls

- Average rank is a lower-is-better metric, contrary to standard accuracy scores.
- Evaluation suites differ by language (SmolLM for English, FineTasks for others), so direct score comparison across languages is invalid; only the aggregated rank is comparable.
- Decontamination removes entire documents flagged by 13-gram overlap, which can disproportionately affect shorter documents or specific domains.

## Evidence (verbatim from paper)

> The average rank is computed across FineTasks performance of 1B-parameter models evaluated after 70B and 119B tokens were consumed.

## Citation

```bibtex
@misc{messmer2025enhancing,
  title={Enhancing Multilingual LLM Pretraining with Model-Based Data Selection},
  author={Messmer et al. (2025)},
  year={2025},
  note={arXiv:2502.10361}
}
```

- arXiv: 2502.10361