La Leaderboard Eval

---
name: la-leaderboard-eval
description: Evaluates LLMs on multilingual proficiency across Spanish varieties and regional languages of Spain and Latin America (Basque, Catalan, Galician). It probes capabilities in natural language inference, reasoning, question answering, summarization, and linguistic acceptability using a resource-efficient few-shot configuration. Use when the user wants to benchmark on La Leaderboard (66 datasets), or asks about evaluating this task. Reports exact-match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.00999
  bibtex_key: grandury2025laleaderboard
  confidence: medium
---

# la-leaderboard-eval

> La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America — Grandury et al. (2025) (arXiv:2507.00999, 2025)

## What this evaluates

Evaluates LLMs on multilingual proficiency across Spanish varieties and regional languages of Spain and Latin America (Basque, Catalan, Galician). It probes capabilities in natural language inference, reasoning, question answering, summarization, and linguistic acceptability using a resource-efficient few-shot configuration.

## Datasets

- **La Leaderboard (66 datasets)** — total ?; splits: test (-1)

## Metrics

- `exact-match` **(primary)** — range: [0, 1]
  - Percentage of predictions that exactly match the gold reference string. The leaderboard also aggregates task-specific accuracy/F1 scores for NLI, QA, classification, and summarization.

## Input / output format

**Input**: Few-shot prompts containing task instructions and examples, followed by the target query in the target language.

**Output**: Model-generated text response.

## Scoring recipe

```python
def score_exact_match(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
    return correct / len(golds)
```

## Common pitfalls

- Assuming translation-based datasets capture sociolinguistic nuance
- Overlooking the difference between base and instruction-tuned model performance on the same task
- Ignoring the few-shot configuration's impact on computational overhead and results

## Evidence (verbatim from paper)

> Each model-task pair was evaluated once, and the raw, pre-normalization results analyzed in this work are publicly available. ... math reasoning yields the lowest results, which could be related to a too strict metric (exact match).

## Citation

```bibtex
@misc{grandury2025laleaderboard,
  title={La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America},
  author={Grandury et al. (2025)},
  year={2025},
  note={arXiv:2507.00999}
}
```

- arXiv: 2507.00999