Multi Prompt Eval

---
name: multi-prompt-eval
description: This protocol evaluates how accurately a statistical estimation method can reconstruct the full performance distribution and specific quantiles of large language models across hundreds of prompt templates, using a fraction of the standard evaluation budget. It probes the robustness of LLM performance metrics against arbitrary prompt selection and measures the efficiency of borrowing strength across prompts and examples. Use when the user wants to benchmark on MMLU, BIG-bench Hard, LMentry, or asks about evaluating this task. Reports Wasserstein-1 distance ($W_1$).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.17202
  bibtex_key: polo2024prompteval
  confidence: high
---

# multi-prompt-eval

> Efficient multi-prompt evaluation of LLMs — Felipe Maia Polo et al. (2024) (arXiv:2405.17202, 2024)

## What this evaluates

This protocol evaluates how accurately a statistical estimation method can reconstruct the full performance distribution and specific quantiles of large language models across hundreds of prompt templates, using a fraction of the standard evaluation budget. It probes the robustness of LLM performance metrics against arbitrary prompt selection and measures the efficiency of borrowing strength across prompts and examples.

## Datasets

- **MMLU** — total 14000; splits: test (14000)
- **BIG-bench Hard** — total 1500; splits: test (1500)
- **LMentry** — total ?; splits: test (-1)

## Metrics

- `Wasserstein-1 distance ($W_1$)` **(primary)** — range: [0, 1] or percent
  - Average absolute difference between sorted true performance scores and sorted estimated performance scores: $W_1(F,\hat{F}) = \frac{1}{I}\sum_{i=1}^{I}|S_{(i)}-\hat{S}_{(i)}|$. Equivalent to the integral of absolute quantile estimation errors over [0,1].
- `Quantile estimation error` — range: [0, 1] or percent
  - Absolute difference between the true quantile $Q(p)$ and the estimated quantile $\hat{Q}(p)$ at probability $p \in \{0.05, 0.25, 0.5, 0.75, 0.95\}$: $|Q(p) - \hat{Q}(p)|$.

## Input / output format

**Input**: LLM prediction scores (or binary correctness) for a subset of benchmark examples evaluated under multiple prompt templates.

**Output**: Estimated performance distribution $\hat{F}$ and estimated quantiles $\hat{Q}(p)$ for specified probabilities $p$.

## Scoring recipe

```python
def compute_w1(true_scores, est_scores):
    true_sorted = sorted(true_scores)
    est_sorted = sorted(est_scores)
    return sum(abs(t - e) for t, e in zip(true_sorted, est_sorted)) / len(true_sorted)

def compute_quantile_error(true_scores, est_scores, p):
    true_q = sorted(true_scores)[int(len(true_scores) * p)]
    est_q = sorted(est_scores)[int(len(est_scores) * p)]
    return abs(true_q - est_q)
```

## Common pitfalls

- Assuming single-prompt evaluation accuracy reflects robustness across prompt variations.
- Confusing the estimation budget (e.g., 200 evaluations) with the full evaluation budget required for ground truth.
- Extreme quantiles (5th/95th) require significantly more evaluations to estimate accurately compared to central quantiles like the median.

## Evidence (verbatim from paper)

> To compare the full performance distribution $F$ and its estimate $\hat{F}$, both defined in §[3], we use the Wasserstein 1-distance which is equivalent to the average quantile estimation error in this case, i.e., $W_{1}(F,\hat{F}) = \int_{0}^{1}|Q(t)-\hat{Q}(t)|\mathrm{d}t=\frac{1}{I}\sum_{i=1}^{I}|S_{(i)}-\hat{S}_{(i)}|$. Second, we estimate some quantiles of interest (e.g., $5/25/50/75/95$-th) for the performance distribution across prompt formats and compare them with the true quantiles, that is, for some $p\in[0,1]$, we use $|Q(p)-\hat{Q}(p)|$ to measure the quality of our estimations.

## Citation

```bibtex
@misc{polo2024prompteval,
  title={Efficient multi-prompt evaluation of LLMs},
  author={Felipe Maia Polo et al. (2024)},
  year={2024},
  note={arXiv:2405.17202}
}
```

- arXiv: 2405.17202