Multi Prompt Llm Eval

---
name: multi-prompt-llm-eval
description: This evaluation protocol probes the robustness of large language models to instruction phrasing by measuring performance across multiple semantically equivalent prompts. It assesses whether model rankings and absolute scores remain stable when the same task is presented with different instruction templates. Use when the user wants to benchmark on LMentry, BIG-bench Lite, BIG-bench Hard, or asks about evaluating this task. Reports exact match evaluation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.00595
  bibtex_key: mizrahi2024multiprompt
  confidence: high
---

# multi-prompt-llm-eval

> State of What Art? A Call for Multi-Prompt LLM Evaluation — Moran Mizrahi et al. (2024) (arXiv:2401.00595, 2024)

## What this evaluates

This evaluation protocol probes the robustness of large language models to instruction phrasing by measuring performance across multiple semantically equivalent prompts. It assesses whether model rankings and absolute scores remain stable when the same task is presented with different instruction templates.

## Datasets

- **LMentry** — total ?; splits: test (-1)
- **BIG-bench Lite** — total ?; splits: test (-1)
- **BIG-bench Hard** — total ?; splits: test (-1)

## Metrics

- `exact match evaluation` **(primary)** — range: [0, 1]
  - Compares the model's generated output string directly to the ground-truth label string. A match yields 1, otherwise 0. Averaged over all instances in the benchmark.

## Input / output format

**Input**: A task instruction (provided in one of several paraphrased templates for LMentry, or a single fixed template for BIG-bench tasks) followed by the task-specific prompt or question.

**Output**: The model's generated text response, which is compared verbatim to the expected answer.

## Scoring recipe

```python
def score(predictions, golds):
    matches = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip() == gold.strip():
            matches += 1
    return matches / len(golds)
```

## Common pitfalls

- Single-prompt evaluations are highly brittle and can misrepresent model capabilities due to instruction phrasing sensitivity.
- Closed API models may manipulate or wrap input prompts, interfering with direct evaluation of instruction templates.
- Exact-match scoring is strict and may penalize semantically correct but syntactically different model outputs.

## Evidence (verbatim from paper)

> We measure performance in the standard manner provided by each benchmark. In LMentry this is done with the official evaluation script, while in Big-Bench we use exact match evaluation. We note that while this evaluation is somewhat strict, we believe that it is also fair and straightforward.

## Citation

```bibtex
@misc{mizrahi2024multiprompt,
  title={State of What Art? A Call for Multi-Prompt LLM Evaluation},
  author={Moran Mizrahi et al. (2024)},
  year={2024},
  note={arXiv:2401.00595}
}
```

- arXiv: 2401.00595