Byol Lrl Eval

---
name: byol-lrl-eval
description: Evaluates LLM capabilities in low- and extreme-low-resource languages (Chichewa, Māori) across reasoning, reading comprehension, factual knowledge, and machine translation. It measures both language-specific adaptation gains and preservation of multilingual/English capabilities. Use when the user wants to benchmark on BYOL Evaluation Benchmarks, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.10804
  bibtex_key: zamir2026byol
  confidence: high
---

# byol-lrl-eval

> BYOL: Bring Your Own Language Into LLMs — Zamir et al. (2026) (arXiv:2601.10804, 2026)

## What this evaluates

Evaluates LLM capabilities in low- and extreme-low-resource languages (Chichewa, Māori) across reasoning, reading comprehension, factual knowledge, and machine translation. It measures both language-specific adaptation gains and preservation of multilingual/English capabilities.

## Datasets

- **BYOL Evaluation Benchmarks** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Proportion of correct predictions out of total instances. Calculated as correct / total.
- `average_score` — range: percent
  - Mean of all benchmark metrics normalized to the [0,1] range before averaging.
- `BLEU` — range: percent
  - Standard n-gram overlap metric for machine translation evaluation.
- `chrF++` — range: percent
  - Character n-gram F-score used for translation evaluation.
- `win-rate` — range: percent
  - Percentage of pairwise comparisons where the model's response is preferred by the LLM-as-a-judge under a forced-choice protocol.

## Input / output format

**Input**: Text prompts or multiple-choice questions in Chichewa, Māori, or English, formatted for the lm-evaluation-harness framework.

**Output**: Model-generated text responses or selected multiple-choice options, evaluated against gold reference answers.

## Scoring recipe

```python
def compute_metrics(predictions, golds, metric_type):
    if metric_type == 'accuracy':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif metric_type in ['BLEU', 'chrF++']:
        return compute_mt_metric(predictions, golds)
    elif metric_type == 'win-rate':
        return sum(1 for p in predictions if p == 'winner') / len(predictions)
    return 0.0

def compute_average_score(benchmark_results):
    normalized = [min(max(r, 0.0), 1.0) for r in benchmark_results.values()]
    return sum(normalized) / len(normalized)
```

## Common pitfalls

- Normalizing disparate metrics (accuracy, BLEU, chrF++) to [0,1] before averaging can obscure task-specific performance differences.
- Machine-translated benchmarks may introduce translation artifacts that confound model capability assessment.
- LLM-as-a-judge forced-choice protocol eliminates ties, potentially inflating win-rates for marginally better models.

## Evidence (verbatim from paper)

> Throughout the paper, we report scores for each benchmark/task using its standard evaluation metric (accuracy, BLEU, chrF++, etc.). The average score (reported as a percentage) is computed by normalizing each metric to the $[0,1]$ range and using chrF++ for the translation task. All translated benchmarks are integrated into the lm-evaluation-harness framework.

## Citation

```bibtex
@misc{zamir2026byol,
  title={BYOL: Bring Your Own Language Into LLMs},
  author={Zamir et al. (2026)},
  year={2026},
  note={arXiv:2601.10804}
}
```

- arXiv: 2601.10804