Finetasks Eval

---
name: finetasks-eval
description: Evaluates the downstream quality of multilingual pretraining corpora by training language models on them and measuring performance on a standardized suite of fine-tuning tasks across Arabic, Hindi, and Turkish. Use when the user wants to benchmark on FineTasks, or asks about evaluating this task. Reports FineTasks scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.18834
  bibtex_key: [REDACTED]
  confidence: high
---

# finetasks-eval

> Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets — Alrashed et al. (2025) (arXiv:2512.18834, 2025)

## What this evaluates

Evaluates the downstream quality of multilingual pretraining corpora by training language models on them and measuring performance on a standardized suite of fine-tuning tasks across Arabic, Hindi, and Turkish.

## Datasets

- **FineTasks** — total ?; splits: test (-1)

## Metrics

- `FineTasks scores` **(primary)** — range: [0, 1]
  - Aggregate score computed as the average of normalized task scores across the FineTasks benchmark suite. Values are reported as floating-point numbers between 0 and 1.

## Input / output format

**Input**: Language models are trained on variants of multilingual pretraining corpora (Arabic, Hindi, Turkish) for up to 30 billion tokens, then evaluated on the FineTasks benchmark suite.

**Output**: Aggregate FineTasks score (a single floating-point number representing average task performance across the benchmark).

## Scoring recipe

```python
scores = []
for task in finetasks_tasks:
    acc = evaluate_model_on_task(task)
    scores.append(normalize(acc))
aggregate_score = sum(scores) / len(scores)
return aggregate_score
```

## Common pitfalls

- Confusing the size of the pretraining corpus (billions of tokens) with the evaluation benchmark size.
- Assuming performance gains are solely due to data quality without controlling for training compute or checkpoint differences.
- Overlooking that FineTasks scores are aggregate averages, which can mask per-language or per-task variations.

## Evidence (verbatim from paper)

> We evaluate MixMinMatch through the training procedure outlined in Section[6.2] on each corpus variant and measure aggregate FineTasks scores as described in Section[6.3].

## Citation

```bibtex
@misc{alrashed2025mixminhash,
  title={Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets},
  author={Alrashed et al. (2025)},
  year={2025},
  note={arXiv:2512.18834}
}
```

- arXiv: 2512.18834