Gpo Prompt Optimization Eval

---
name: gpo-prompt-optimization-eval
description: Evaluates the effectiveness of LLM-based prompt optimizers across complex reasoning, knowledge-intensive, and common NLP tasks. It measures how much optimized prompts improve model performance compared to baseline prompts and other optimization methods. Use when the user wants to benchmark on Big-Bench Hard (BBH), GSM8K, MMLU, WSC, WebNLG, or asks about evaluating this task. Reports average accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.17564
  bibtex_key: tang2024gpo
  confidence: high
---

# gpo-prompt-optimization-eval

> Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers — Tang et al. (2024) (arXiv:2402.17564, 2024)

## What this evaluates

Evaluates the effectiveness of LLM-based prompt optimizers across complex reasoning, knowledge-intensive, and common NLP tasks. It measures how much optimized prompts improve model performance compared to baseline prompts and other optimization methods.

## Datasets

- **Big-Bench Hard (BBH)** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **WSC** — total ?; splits: test (-1)
- **WebNLG** — total ?; splits: test (-1)

## Metrics

- `average accuracy` **(primary)** — range: [0, 100] percent
  - Calculated as the number of correct predictions divided by the total number of instances, multiplied by 100. Reported as the average across all subtasks for BBH and MMLU, and directly for GSM8K.
- `ROUGE-L` — range: [0, 1]
  - Computes the longest common subsequence between the reference and generated text to measure overlap. Used for WSC and WebNLG following Wang et al. (2022).

## Input / output format

**Input**: Task prompt (generated by the optimizer or baseline) concatenated with the input question or example from the dataset.

**Output**: Model-generated text or discrete answer corresponding to the dataset's expected output format.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if normalize_answer(p) == normalize_answer(g))
    return (correct / len(golds)) * 100

def compute_rouge_l(predictions, references):
    scores = [rouge_l_score(p, r) for p, r in zip(predictions, references)]
    return sum(scores) / len(scores)
```

## Common pitfalls

- Experiments use sampled subsets of each dataset due to computational limitations, not the full benchmarks.
- Results are averaged over three independent runs, but standard deviations or confidence intervals are not reported in the main tables.
- ROUGE-L is used for WSC and WebNLG, which may not align perfectly with exact-match accuracy used for reasoning tasks.

## Evidence (verbatim from paper)

> Evaluation Metrics. We report the average accuracy of all the subtasks for BBH and MMLU following *Suzgun et al. ([2023](#bib.bib31 ""))* and *Hendrycks et al. ([2021](#bib.bib13 ""))*, accuracy for GSM8K following *Cobbe et al. ([2021](#bib.bib5 ""))*, ROUGE-L*Lin ([2004](#bib.bib20 ""))* for WSC and WebNLG following *Wang et al. ([2022](#bib.bib33 ""))*.

## Citation

```bibtex
@misc{tang2024gpo,
  title={Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers},
  author={Tang et al. (2024)},
  year={2024},
  note={arXiv:2402.17564}
}
```

- arXiv: 2402.17564