Structured Prompting Eval

---
name: structured-prompting-eval
description: Evaluates how different prompting strategies (baseline, zero-shot, chain-of-thought, and automated optimizers) affect the accuracy, ranking stability, and variance of language model performance across multiple knowledge and reasoning benchmarks. Use when the user wants to benchmark on MMLU-Pro, GSM8K, MedCalc-Bench, GPQA, HeadQA, MedBullets, Medec, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.20836
  bibtex_key: aali2025structuredprompting
  confidence: high
---

# structured-prompting-eval

> Structured Prompting Enables More Robust Evaluation of Language Models — Aali et al. (2025) (arXiv:2511.20836, 2025)

## What this evaluates

Evaluates how different prompting strategies (baseline, zero-shot, chain-of-thought, and automated optimizers) affect the accuracy, ranking stability, and variance of language model performance across multiple knowledge and reasoning benchmarks.

## Datasets

- **MMLU-Pro** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **MedCalc-Bench** — total ?; splits: test (-1)
- **GPQA** — total ?; splits: test (-1)
- **HeadQA** — total ?; splits: test (-1)
- **MedBullets** — total ?; splits: test (-1)
- **Medec** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of instances. Macro-averaged across benchmarks when reporting aggregate performance.

## Input / output format

**Input**: Task-specific question or prompt (e.g., multiple-choice question, patient note) prepended with a structured prompt template containing instructions and/or few-shot demonstrations.

**Output**: Final answer string, optionally preceded by a chain-of-thought reasoning trace. Output length capped at <200 tokens.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip())
    return (correct / len(gold)) * 100

def compute_mean_rank(model_accuracies):
    ranks = sorted(model_accuracies.items(), key=lambda x: x[1], reverse=True)
    return {name: rank + 1 for rank, (name, _) in enumerate(ranks)}
```

## Common pitfalls

- Prompt design significantly alters leaderboard rankings; a model's rank is not invariant to the prompting method used.
- Non-reasoning baselines (Zero-Shot Predict) underperform by ~4% on average compared to CoT variants, which can mask true capability differences.
- Optimization costs (BFRS, MIPROv2) are one-time amortized expenses, while inference costs scale with prompt token length.

## Evidence (verbatim from paper)

> Structured prompting methods (Zero-Shot CoT, BFRS, MIPROv2) consistently improve over the HELM baseline (Table [3]). On average, LMs gain $+$4% in absolute accuracy.

## Citation

```bibtex
@misc{aali2025structuredprompting,
  title={Structured Prompting Enables More Robust Evaluation of Language Models},
  author={Aali et al. (2025)},
  year={2025},
  note={arXiv:2511.20836}
}
```

- arXiv: 2511.20836