Llama4 Benchmark Eval

---
name: llama4-benchmark-eval
description: Evaluates the language understanding, reasoning, coding, multilingual, and multimodal capabilities of Llama 4 models across a standardized suite of academic and industry benchmarks. It measures performance on text-only, code, and vision-language tasks using few-shot or zero-shot prompting protocols. Use when the user wants to benchmark on MMLU, MMLU-Pro, MATH, MBPP, LiveCodeBench, GPQA Diamond, ChartQA, DocVQA, MMMU, MTOB, or asks about evaluating this task. Reports macro_avg/acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.11659
  bibtex_key: adcock2026llama4
  confidence: high
---

# llama4-benchmark-eval

> The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes — Adcock et al. (2026) (arXiv:2601.11659, 2026)

## What this evaluates

Evaluates the language understanding, reasoning, coding, multilingual, and multimodal capabilities of Llama 4 models across a standardized suite of academic and industry benchmarks. It measures performance on text-only, code, and vision-language tasks using few-shot or zero-shot prompting protocols.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **MMLU-Pro** — total ?; splits: test (-1)
- **MATH** — total ?; splits: test (-1)
- **MBPP** — total ?; splits: test (-1)
- **LiveCodeBench** — total ?; splits: test (-1)
- **GPQA Diamond** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MTOB** — total ?; splits: test (-1)

## Metrics

- `macro_avg/acc` **(primary)** — range: [0, 1]
  - Macro-averaged accuracy across all categories or tasks. Computed as the mean of per-category accuracy scores.
- `em` — range: [0, 1]
  - Exact-match accuracy. Returns 1 if the model's predicted answer exactly matches the ground truth, 0 otherwise.
- `pass@1` — range: [0, 1]
  - Probability that the first generated solution passes all provided unit tests or satisfies the problem constraints.
- `relaxed_accuracy` — range: [0, 1]
  - Accuracy metric for chart understanding that allows for minor formatting or textual variations in the answer while preserving semantic correctness.
- `ANLS` — range: [0, 1]
  - Average Normalized Levenshtein Similarity. Computes the Levenshtein distance between predicted and ground truth text, normalized by the length of the ground truth, averaged across samples.
- `chrF` — range: [0, 100]
  - Character n-gram F-score. Measures the harmonic mean of character n-gram precision and recall between the generated translation and the reference, commonly used for long-context machine translation evaluation.

## Input / output format

**Input**: Text prompts with optional images or code snippets, formatted with few-shot examples where specified (e.g., 5-shot for MMLU/MMLU-Pro, 4-shot for MATH, 3-shot for MBPP, 1-shot for TyDiQA, 0-shot for others). Long-context tasks use full-book or half-book documents up to 128K context window.

**Output**: Natural language answers, numerical values, or executable code blocks, depending on the benchmark. Multimodal tasks require text responses grounded in provided images.

## Scoring recipe

```python
def compute_metrics(predictions, golds, metric_name):
    if metric_name == 'macro_avg/acc':
        return np.mean([np.mean(p == g) for p, g in zip(predictions, golds)])
    elif metric_name == 'em':
        return np.mean([1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)])
    elif metric_name == 'pass@1':
        return np.mean([1.0 if passes_unit_test(p) else 0.0 for p in predictions])
    elif metric_name == 'relaxed_accuracy':
        return np.mean([1.0 if is_semantically_equivalent(p, g) else 0.0 for p, g in zip(predictions, golds)])
    elif metric_name == 'ANLS':
        return np.mean([levenshtein_sim(p, g) for p, g in zip(predictions, golds)])
    elif metric_name == 'chrF':
        return compute_chrf_score(predictions, golds)
    return 0.0
```

## Common pitfalls

- Results are exclusively reported on bf16 precision; mixed precision or quantized evaluations are not included and may differ significantly.
- Multimodal baselines are excluded from comparison due to lack of multimodal support, making direct performance benchmarking against prior Llama versions impossible for vision tasks.
- Long-context evaluation uses MTOB with a fixed 128K context window, which may not reflect true infinite-context capabilities or retrieval accuracy beyond that limit.

## Evidence (verbatim from paper)

> Long-context evaluation is presented via MTOB with chrF scores in two translation directions, with the model card explicitly contrasting against a 128K context baseline

## Citation

```bibtex
@misc{adcock2026llama4,
  title={The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes},
  author={Adcock et al. (2026)},
  year={2026},
  note={arXiv:2601.11659}
}
```

- arXiv: 2601.11659