Aide Benchmark Eval

---
name: aide-benchmark-eval
description: Evaluates the performance of LLMs fine-tuned on synthetic data generated by AIDE across a suite of standard knowledge and reasoning benchmarks. It probes zero-shot and few-shot generalization capabilities compared to models fine-tuned on human-curated gold data. Use when the user wants to benchmark on MMLU, FinBen, ARC-Challenge, GSM8K, TruthfulQA, MedQA, BIG-Bench, or asks about evaluating this task. Reports zero-shot accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.06136
  bibtex_key: li2024aide
  confidence: high
---

# aide-benchmark-eval

> AIDE: Attribute-Guided MultI-Hop Data Expansion for Data Scarcity in Task-Specific Fine-tuning — Jiayu Li et al. (arXiv:2412.06136, 2024)

## What this evaluates

Evaluates the performance of LLMs fine-tuned on synthetic data generated by AIDE across a suite of standard knowledge and reasoning benchmarks. It probes zero-shot and few-shot generalization capabilities compared to models fine-tuned on human-curated gold data.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **FinBen** — total ?; splits: test (-1)
- **ARC-Challenge** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **TruthfulQA** — total ?; splits: test (-1)
- **MedQA** — total ?; splits: test (-1)
- **BIG-Bench** — total ?; splits: test (-1)

## Metrics

- `zero-shot accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers across all benchmark tasks. For GSM8K, the paper reports 8-shot maj@8 performance instead.

## Input / output format

**Input**: Benchmark-specific prompts. Most tasks use zero-shot prompts. GSM8K uses 8-shot prompts with in-context examples.

**Output**: Model-generated answer strings or multiple-choice selections.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100

def compute_maj_at_8(predictions_8shot):
    # predictions_8shot is a list of lists, each inner list has 8 generations
    maj_votes = [Counter(p).most_common(1)[0][0] for p in predictions_8shot]
    return maj_votes
```

## Common pitfalls

- GSM8K uses a different evaluation protocol (8-shot maj@8) than the zero-shot accuracy used for other benchmarks.
- The paper reports average performance across benchmarks, which can mask task-specific variance.
- Self-BLEU is used for diversity analysis but is not the primary performance metric.

## Evidence (verbatim from paper)

> We evaluated all models using zero-shot accuracy as the primary metric on the benchmarks. For GSM8K, we report 8-shot maj@8 performance using prompts from Wang et al. (2023).

## Citation

```bibtex
@misc{li2024aide,
  title={AIDE: Attribute-Guided MultI-Hop Data Expansion for Data Scarcity in Task-Specific Fine-tuning},
  author={Jiayu Li et al.},
  year={2024},
  note={arXiv:2412.06136}
}
```

- arXiv: 2412.06136