Meta Metrics Inference Bench

---
name: meta-metrics-inference-bench
description: Evaluates the efficiency and representational fidelity of LLM inference benchmarking methodologies by quantifying how well a reduced set of experimental parameters can accurately predict system performance compared to exhaustive testing. It measures the trade-off between computational cost and the accuracy of projected latency and throughput metrics. Use when the user has predictions and gold and needs to compute efficiency_metric.
metadata:
  skill_kind: metric
  source_arxiv: 2508.10251
  bibtex_key: salaria2025metametrics
  confidence: high
---

# meta-metrics-inference-bench

> Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking — Salaria et al. (2025) (arXiv:2508.10251, 2025)

## What this evaluates

Evaluates the efficiency and representational fidelity of LLM inference benchmarking methodologies by quantifying how well a reduced set of experimental parameters can accurately predict system performance compared to exhaustive testing. It measures the trade-off between computational cost and the accuracy of projected latency and throughput metrics.

## Datasets

- **Synthetic Inference Workloads** — total ?; splits: (unstated)

## Metrics

- `ttft` — range: milliseconds
  - Time to First Token: measures the latency from sending a prompt to receiving the first generated token. Independent of output length.
- `itl` — range: milliseconds
  - Inter-Token Latency: measures the average time between subsequent generated tokens during autoregressive decoding. Varies with output length and batch size.
- `efficiency_metric` **(primary)** — range: other (ratio/factor)
  - E(G, P) = (1 - Δ(G, P)) / (C_P / C_G), where Δ is the global accuracy factor (e.g., average normalized relative difference) between ground truth measurements G and projected measurements P, and C represents total experimental cost.

## Input / output format

**Input**: Control parameters: model architecture, precision, parallelism configuration, inference backend, input sequence length, output generation length, and batch size.

**Output**: Performance measurements: Time to First Token (TTFT), Inter-Token Latency (ITL), Throughput (THP in tokens/sec), and Cost per million tokens ($/M).

## Scoring recipe

```python
def compute_efficiency_metric(G_metrics, P_metrics, C_G, C_P):
    delta_values = []
    for key in G_metrics:
        f_G = G_metrics[key]
        f_P = P_metrics[key]
        delta = abs(f_G - f_P) / (f_G + f_P)
        delta_values.append(delta)
    delta_global = sum(delta_values) / len(delta_values)
    efficiency = (1 - delta_global) / (C_P / C_G)
    return efficiency
```

## Common pitfalls

- Assuming linear interpolation accurately captures non-linear GPU saturation effects at high batch sizes without empirical validation.
- Confusing system-level inference metrics (TTFT, ITL, throughput) with model accuracy/quality metrics, which are explicitly out of scope for this framework.
- Ignoring hardware-specific memory constraints when scaling batch sizes or comparing tensor parallelism configurations across different GPU counts.

## Evidence (verbatim from paper)

> A simple efficiency metric E can be defined as: E(G, P) = (1 - Δ(G, P)) / (C_P / C_G)

## Citation

```bibtex
@misc{salaria2025metametrics,
  title={Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking},
  author={Salaria et al. (2025)},
  year={2025},
  note={arXiv:2508.10251}
}
```

- arXiv: 2508.10251