Ernie 5.0 Eval

---
name: ernie-5.0-eval
description: Evaluates a trillion-parameter multimodal foundation model across text, vision, audio, and generation tasks to measure factual knowledge, reasoning, coding, instruction following, and agent capabilities. Use when the user wants to benchmark on PreciseWikiQA, MMLU-Pro, MATH, LiveCodeBench, MMMU-Pro, MathVista, GenEval, VBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.04705
  bibtex_key: wang2026ernie5.0
  confidence: high
---

# ernie-5.0-eval

> ERNIE 5.0 Technical Report — Haifeng Wang et al. (2026) (arXiv:2602.04705, 2026)

## What this evaluates

Evaluates a trillion-parameter multimodal foundation model across text, vision, audio, and generation tasks to measure factual knowledge, reasoning, coding, instruction following, and agent capabilities.

## Datasets

- **PreciseWikiQA** — total ?; splits: test (-1)
- **MMLU-Pro** — total ?; splits: test (-1)
- **MATH** — total ?; splits: test (-1)
- **LiveCodeBench** — total ?; splits: test (-1)
- **MMMU-Pro** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **GenEval** — total ?; splits: test (-1)
- **VBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers or task-specific scores normalized to [0, 100]. Different benchmarks use underlying metrics such as exact match, F1, or LLM-as-judge, but all results are reported as percentages.

## Input / output format

**Input**: Task-specific prompts including text questions, images, audio, or video, often with few-shot examples (0 to 10 shots) and explicit instructions.

**Output**: Model-generated text responses, code, or generated images/videos depending on the benchmark task.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Shot counts vary widely across benchmarks (0-shot to 10-shot), and few-shot prompting strategies are not standardized.
- Different benchmarks use different underlying evaluation metrics (exact match, F1, automated generation metrics), but results are uniformly reported as percentages, obscuring task-specific nuances.
- Some generation benchmarks (e.g., GenEval, VBench) rely on automated metrics that may not fully correlate with human preference or real-world quality.

## Evidence (verbatim from paper)

> Table 1 summarizes the pre-training results of ERNIE 5.0 in comparison with strong open-source baselines on a diverse set of text benchmarks. Across these benchmarks, ERNIE 5.0 exhibits consistently strong and well-balanced performance in knowledge, reasoning, mathematics, coding, and multilingual tasks:

## Citation

```bibtex
@misc{wang2026ernie5.0,
  title={ERNIE 5.0 Technical Report},
  author={Haifeng Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.04705}
}
```

- arXiv: 2602.04705