Gamayun Eval

---
name: gamayun-eval
description: Evaluates multilingual LLM capabilities across general knowledge, reasoning, mathematics, and cultural understanding in English, Russian, and other languages. It probes zero-shot and few-shot performance on standardized benchmarks and custom cultural knowledge tests. Use when the user wants to benchmark on MMLU, GSM8K, MERA, RuBIN, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.21580
  bibtex_key: podolskiy2025gamayun
  confidence: high
---

# gamayun-eval

> Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM — Podolskiy et al. (2025) (arXiv:2512.21580, 2025)

## What this evaluates

Evaluates multilingual LLM capabilities across general knowledge, reasoning, mathematics, and cultural understanding in English, Russian, and other languages. It probes zero-shot and few-shot performance on standardized benchmarks and custom cultural knowledge tests.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **MERA** — total ?; splits: test (-1)
- **RuBIN** — total 5000; splits: test (5000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice or closed-form questions. Calculated as (correct predictions / total instances) * 100.
- `MERA score` — range: percent
  - Average performance across 15 tasks assessing different LLM abilities, using closed test subsets.
- `RuBIN score` — range: percent
  - Accuracy on a 5,000-item multiple-choice benchmark covering culturally specific Russian topics (proverbs, riddles, songs, films).

## Input / output format

**Input**: Multiple-choice questions or open-ended prompts, typically provided with few-shot examples (e.g., 5-shot) or zero-shot instructions.

**Output**: Model generates text completion or selects the correct option from provided choices.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if normalize(p) == normalize(g))
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Using different shot counts than specified (e.g., 5-shot vs 0-shot) can drastically change scores.
- MERA and RuBIN are not publicly available or are internal/closed benchmarks, making direct replication difficult.
- Aggregating scores across languages/tasks without weighting can mask performance distributional skew.

## Evidence (verbatim from paper)

> We use zero-shot evaluation in all cases unless explicitly stated otherwise. ... The MERA score is calculated as the average performance across 15 tasks assessing different LLM abilities, using closed test subsets.

## Citation

```bibtex
@misc{podolskiy2025gamayun,
  title={Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM},
  author={Podolskiy et al. (2025)},
  year={2025},
  note={arXiv:2512.21580}
}
```

- arXiv: 2512.21580