Macedonian Benchmarks Eval

---
name: macedonian-benchmarks-eval
description: Evaluates a low-resource language model's capability on standard commonsense reasoning, reading comprehension, and factual knowledge tasks adapted to Macedonian. It measures how well continued pretraining and instruction tuning improve performance on these benchmarks compared to multilingual baselines. Use when the user wants to benchmark on Macedonian Benchmarks (ARC Easy, ARC Challenge, BoolQ, HellaSwag, OpenBookQA, PIQA, WinoGrande), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.09560
  bibtex_key: krsteski2025towards
  confidence: high
---

# macedonian-benchmarks-eval

> Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language — Krsteski et al. (2025) (arXiv:2506.09560, 2025)

## What this evaluates

Evaluates a low-resource language model's capability on standard commonsense reasoning, reading comprehension, and factual knowledge tasks adapted to Macedonian. It measures how well continued pretraining and instruction tuning improve performance on these benchmarks compared to multilingual baselines.

## Datasets

- **Macedonian Benchmarks (ARC Easy, ARC Challenge, BoolQ, HellaSwag, OpenBookQA, PIQA, WinoGrande)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted answers out of the total number of instances in the benchmark.

## Input / output format

**Input**: Multiple-choice questions in Macedonian.

**Output**: Model generates a predicted answer choice or text response.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Benchmarks are standard English datasets adapted to Macedonian; performance may not directly transfer to English or other languages.
- Qualitative human evaluation uses only 10 prompts, which may not represent full model capability or generalizability.
- Average score across 7 benchmarks is used as a single headline metric, masking task-specific strengths and weaknesses.

## Evidence (verbatim from paper)

> Finally, our model represents a significant improvement compared to the previous best Macedonian model, MKLLM, achieving higher accuracy across six out of seven benchmarks.

## Citation

```bibtex
@misc{krsteski2025towards,
  title={Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language},
  author={Krsteski et al. (2025)},
  year={2025},
  note={arXiv:2506.09560}
}
```

- arXiv: 2506.09560