Flexbench Eval

---
name: flexbench-eval
description: This evaluation probes the inference throughput and generation quality of LLMs across diverse hardware and software configurations. It also tests a predictive modeling framework designed to optimize system co-design by forecasting performance metrics based on model and hardware features. Use when the user wants to benchmark on OpenOrca, Open MLPerf Dataset, or asks about evaluating this task. Reports Tokens/s.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.11413
  bibtex_key: [REDACTED]
  confidence: medium
---

# flexbench-eval

> Framing AI System Benchmarking as a Learning Task: FlexBench and the Open MLPerf Dataset — Fursin et al. (2025) (arXiv:2509.11413, 2025)

## What this evaluates

This evaluation probes the inference throughput and generation quality of LLMs across diverse hardware and software configurations. It also tests a predictive modeling framework designed to optimize system co-design by forecasting performance metrics based on model and hardware features.

## Datasets

- **OpenOrca** — total ?; splits: (unstated)
- **Open MLPerf Dataset** — total ?; splits: (unstated); repo https://github.com/mlcommons/ck

## Metrics

- `ROUGE1/2/L accuracy` — range: [0, 1]
  - Standard ROUGE metrics computing recall/precision of overlapping n-grams (1-gram, 2-gram) and longest common subsequence (L) between generated text and reference answers.
- `Tokens/s` **(primary)** — range: other
  - Inference throughput measured as the total number of generated tokens divided by the total inference time, reported per accelerator and overall.

## Input / output format

**Input**: LLM prompts from the OpenOrca dataset, along with system configuration parameters (model architecture, parameter count, weight data type, framework, OS, accelerator type/count, CPU specs).

**Output**: Generated text completions for ROUGE scoring, and system performance logs reporting throughput (Tokens/s) and latency metrics.

## Scoring recipe

```python
def score(predictions, references, throughput_tokens, time_seconds):
    rouge_scores = {}
    for metric in ['ROUGE1', 'ROUGE2', 'ROUGEL']:
        rouge_scores[metric] = compute_rouge(predictions, references, metric)
    throughput = throughput_tokens / time_seconds
    return {'accuracy': rouge_scores, 'throughput': throughput}
```

## Common pitfalls

- Heterogeneous hardware and software stacks require careful normalization to ensure fair cross-system comparisons.
- ROUGE metrics may poorly correlate with human-judged quality or factual correctness for complex reasoning tasks.
- Predictive models depend on cleaned, standardized features that may not capture all real-world system bottlenecks or dynamic workloads.

## Evidence (verbatim from paper)

> We validated our approach in the MLPerf Inference 5.0 submission by benchmarking several non-MLPerf LLMs (e.g., DeepSeek R1 Distill LLaMA 8B, LLaMA 3.3) on the OpenOrca dataset using commodity servers with NVIDIA H100 GPUs. ... A sample entry is shown below: {"metrics.accuracy": "ROUGE1: 30.6202 ROUGE2: 13.9221 ROUGEL: 18.9101 TOKENS_PER_SAMPLE: 581.8","metrics.result": 2631.93, "metrics.result_per_accelerator": 2631.93,"metrics.units": "Tokens/s", ...}

## Citation

```bibtex
@misc{fursin2025flexbench,
  title={Framing AI System Benchmarking as a Learning Task: FlexBench and the Open MLPerf Dataset},
  author={Fursin et al. (2025)},
  year={2025},
  note={arXiv:2509.11413}
}
```

- arXiv: 2509.11413