Mmlu Bbh Gsm8k Eval

---
name: mmlu-bbh-gsm8k-eval
description: Evaluates how fine-tuning large language models on syntactically or semantically perturbed instructions impacts downstream generalization across factual knowledge, complex reasoning, and mathematical problem-solving. It also measures potential side effects on model toxicity and truthfulness under varying noise levels during both training and evaluation. Use when the user wants to benchmark on MMLU, BBH, GSM8K, ToxiGen, TruthfulQA, or asks about evaluating this task. Reports average test accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.03528
  bibtex_key: alajrami2025finetuning
  confidence: high
---

# mmlu-bbh-gsm8k-eval

> Fine-Tuning on Noisy Instructions: Effects on Generalization and Performance — Alajrami et al. (2025) (arXiv:2510.03528, 2025)

## What this evaluates

Evaluates how fine-tuning large language models on syntactically or semantically perturbed instructions impacts downstream generalization across factual knowledge, complex reasoning, and mathematical problem-solving. It also measures potential side effects on model toxicity and truthfulness under varying noise levels during both training and evaluation.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **BBH** — total ?; splits: test (-1)
- **GSM8K** — total 8500; splits: test (-1)
- **ToxiGen** — total ?; splits: test (-1)
- **TruthfulQA** — total ?; splits: test (-1)

## Metrics

- `average test accuracy` **(primary)** — range: percent
  - Average test accuracy across 57 subjects, computed as the proportion of correctly answered multiple-choice questions in 0-shot and 5-shot settings.
- `average exact match` — range: percent
  - Average exact match across 23 sub-tasks, comparing model output to ground truth answers using direct prompting or chain-of-thought with 3 in-context examples.
- `exact match` — range: percent
  - Exact match between model-generated final answer and ground truth for grade-school math word problems, evaluated with direct prompting or chain-of-thought using 8 in-context examples.
- `toxicity rate` — range: percent
  - Percentage of generated outputs classified as toxic or hate speech by a fine-tuned RoBERTa classifier across demographic prompts.
- `truthfulness score` — range: other
  - Truthfulness and informativeness scores measured by off-the-shelf Llama-2 (7B) judge models, assessing the model's ability to avoid known falsehoods while remaining informative.

## Input / output format

**Input**: Multiple-choice questions (MMLU), reasoning/math word problems (BBH, GSM8K), demographic prompts (ToxiGen), and misconception prompts (TruthfulQA), provided with 0-shot, 5-shot, or few-shot chain-of-thought examples depending on the benchmark.

**Output**: Text responses generated by the model, which are either directly compared to ground truth for exact match/accuracy, or passed to specialized judge models/classifiers for toxicity and truthfulness scoring.

## Scoring recipe

```python
def compute_metrics(predictions, golds, metric_type):
    if metric_type in ['accuracy', 'exact_match']:
        correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
        return correct / len(golds) * 100
    elif metric_type == 'toxicity':
        toxic_count = sum(1 for p in predictions if classifier.predict(p) == 'toxic')
        return toxic_count / len(predictions) * 100
    elif metric_type == 'truthfulness':
        scores = [judge_model.score(p) for p in predictions]
        return sum(scores) / len(scores)
```

## Common pitfalls

- Confusing the perturbation rate used during fine-tuning with the perturbation rate used during evaluation; both are independently varied at 0%, 25%, 50%, 75%, and 100%.
- Assuming identical few-shot prompting across benchmarks; BBH uses 3 in-context examples while GSM8K uses 8, and both support direct vs. CoT prompting.
- Ignoring reported standard deviations over three runs when claiming performance improvements, as results vary across random seeds.

## Evidence (verbatim from paper)

> We assess downstream performance using: Massive Multitask Language Understanding (MMLU; Hendrycks et al. [2021]): MMLU evaluates a model’s factual knowledge and reasoning across 57 subjects, ranging from elementary to professional-level difficulty, using multiple-choice questions. We follow the original MMLU setup, evaluating in 0-shot and 5-shot settings, and report average test accuracy. Big-Bench Hard (BBH; Suzgun et al. [2022]): A challenging subset of 23 tasks from the original BIG-Bench (Srivastava et al., [2023]), aimed at evaluating advanced reasoning in language models. We assess both direct prompting and chain-of-thought (CoT) Wei et al. ([2022b]), using official prompts with three in-context examples, and report average exact match across sub-tasks. Grade School Math (GSM8K; Cobbe et al. [2021]): A benchmark of 8.5K grade school-level word problems for testing multi-step mathematical reasoning in language models. We evaluate with direct prompting and CoT using eight in-context few-shot examples, and we report the exact match.

## Citation

```bibtex
@misc{alajrami2025finetuning,
  title={Fine-Tuning on Noisy Instructions: Effects on Generalization and Performance},
  author={Alajrami et al. (2025)},
  year={2025},
  note={arXiv:2510.03528}
}
```

- arXiv: 2510.03528