Benchmax Eval

---
name: benchmax-eval
description: BenchMAX evaluates the language-agnostic capabilities of large language models across 17 languages, including non-Latin scripts. It probes instruction following, reasoning, code generation, long-context modeling, tool use, and translation through a rigorously translated and human-post-edited pipeline. Use when the user wants to benchmark on BenchMAX, or asks about evaluating this task. Reports evaluation metrics.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.07346
  bibtex_key: huang2025benchmax
  confidence: high
---

# benchmax-eval

> BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models — Xu Huang et al. (arXiv:2502.07346, 2025)

## What this evaluates

BenchMAX evaluates the language-agnostic capabilities of large language models across 17 languages, including non-Latin scripts. It probes instruction following, reasoning, code generation, long-context modeling, tool use, and translation through a rigorously translated and human-post-edited pipeline.

## Datasets

- **BenchMAX** — total ?; splits: test (-1); repo https://github.com/CONE-MT/BenchMAX.git

## Metrics

- `evaluation metrics` **(primary)** — range: percent
  - Percentage of instances where the model's output correctly satisfies task-specific constraints, matches the ground truth answer, or passes automated execution tests. Evaluated via rule-based verifiers, model-based judges (GEMBA-SQM/Qwen2.5-72B), or exact match/execution.

## Input / output format

**Input**: Multilingual prompts (translated from English source datasets) containing task-specific instructions, constraints, code stubs, or long-context documents, provided in one of 17 target languages.

**Output**: Model-generated text, code, or answers in the corresponding target language, adhering to any specified constraints (e.g., length, forbidden keywords).

## Scoring recipe

```python
def score(predictions, gold, task_type):
    if task_type == 'rule_based_if':
        return check_constraints(predictions, gold['kwargs'])
    elif task_type == 'code':
        return run_tests(predictions, gold['tests'])
    elif task_type == 'reasoning':
        return exact_match(predictions, gold['answer'])
    elif task_type == 'translation':
        return judge_quality(predictions, gold['reference'])
    return 0.0
```

## Common pitfalls

- Constraint extraction failure: Special symbols or placeholders used during translation may be lost or mangled, causing rule-based verifiers to incorrectly flag valid outputs.
- Position bias in judging: Human annotators and LLM judges tend to favor the first or last position in pairwise comparisons, requiring shuffling and multiple battles to debias.
- Assuming direct transferability: English evaluation protocols do not account for script-specific tokenization or cultural/domain nuances in non-Latin languages, leading to inflated or deflated scores if not adapted.

## Evidence (verbatim from paper)

> The information of the seleted datasets, sample sizes, and evaluation metrics is provided in Table[3]. More details can be found in Appendix[A].

## Citation

```bibtex
@misc{huang2025benchmax,
  title={BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models},
  author={Xu Huang et al.},
  year={2025},
  note={arXiv:2502.07346}
}
```

- arXiv: 2502.07346