Main Instruction Tuning Eval

---
name: main-instruction-tuning-eval
description: Evaluates the instruction-following capability, output preference quality, and general reasoning performance of fine-tuned LLMs. It probes how well models adhere to explicit constraints, generate preferred responses relative to a baseline, and solve standard academic benchmarks. Use when the user wants to benchmark on AlpacaEval, IFEval, ARC, HellaSwag, Winogrande, MMLU, TruthfulQA, or asks about evaluating this task. Reports AlpacaEval.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.12913
  bibtex_key: yang2025main
  confidence: high
---

# main-instruction-tuning-eval

> MAIN: Mutual Alignment Is Necessary for instruction tuning — Yang et al. (2025) (arXiv:2504.12913, 2025)

## What this evaluates

Evaluates the instruction-following capability, output preference quality, and general reasoning performance of fine-tuned LLMs. It probes how well models adhere to explicit constraints, generate preferred responses relative to a baseline, and solve standard academic benchmarks.

## Datasets

- **AlpacaEval** — total 805; splits: test (805)
- **IFEval** — total ?; splits: test (-1)
- **ARC** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **Winogrande** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **TruthfulQA** — total ?; splits: test (-1)

## Metrics

- `AlpacaEval` **(primary)** — range: percent
  - Percentage of pairwise comparisons where the model's output is judged better than text-davinci-003 by GPT-4. Calculated as (number of wins / total instructions) * 100.
- `IFEval` — range: percent
  - Accuracy across four variants: Prompt-level Strict (P-S), Instruction-level Strict (I-S), Prompt-level Loose (P-L), and Instruction-level Loose (I-L). Measures whether generated outputs satisfy explicit formatting and constraint instructions.
- `OpenLLM Leaderboard accuracy` — range: percent
  - Standard accuracy scores on ARC, HellaSwag, Winogrande, MMLU, and TruthfulQA, evaluated using the Language Model Evaluation Harness.

## Input / output format

**Input**: Instruction or prompt text; for IFEval, prompts include explicit formatting/constraint instructions; for OpenLLM tasks, multiple-choice or open-ended questions.

**Output**: Model-generated text response.

## Scoring recipe

```python
def score_alpaca_eval(predictions, baseline_outputs):
    wins = 0
    for pred, base in zip(predictions, baseline_outputs):
        if gpt4_judge(pred, base) == 'win':
            wins += 1
    return (wins / len(predictions)) * 100

def score_ifeval(predictions, prompts):
    accuracies = []
    for pred, prompt in zip(predictions, prompts):
        accuracies.append(check_constraints(pred, prompt))
    return sum(accuracies) / len(accuracies) * 100
```

## Common pitfalls

- AlpacaEval win rates are relative to text-davinci-003, not gold references, so scores reflect preference over a specific baseline rather than absolute quality.
- IFEval reports four distinct strict/loose variants; reporting only one metric can significantly misrepresent instruction-following capability.
- OpenLLM tasks are evaluated via the Language Model Evaluation Harness, which may apply specific prompting or few-shot templates that differ from standard zero-shot setups.

## Evidence (verbatim from paper)

> We assess output preference using 805 instructions from the AlpacaEval dataset. Model outputs are compared against text-davinci-003 in a pairwise setting, with GPT-4-based judgments determining win rates. Instruction-following ability is evaluated with IFEval, which reports accuracy across four metrics: Prompt-level Strict (P-S), Instruction-level Strict (I-S), Prompt-level Loose (P-L), Instruction-level Loose (I-L) ensuring a comprehensive assessment of instruction adherence.

## Citation

```bibtex
@misc{yang2025main,
  title={MAIN: Mutual Alignment Is Necessary for instruction tuning},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2504.12913}
}
```

- arXiv: 2504.12913