Lm Eval Harness Benchmarks Eval

---
name: lm-eval-harness-benchmarks-eval
description: Evaluates generative language models on a suite of multiple-choice and open-ended benchmarks covering reasoning, commonsense, multitask proficiency, and truthfulness. It measures accuracy across diverse domains to assess generalization and the impact of data combination strategies. Use when the user wants to benchmark on AI2 Reasoning Challenge (ARC), HellaSwag, MMLU, TruthfulQA, BigBench, HumanEval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.10818
  bibtex_key: shen2023slimpajamadc
  confidence: high
---

# lm-eval-harness-benchmarks-eval

> SlimPajama-DC: Understanding Data Combinations for LLM Training — Zhiqiang Shen et al. (2023) (arXiv:2309.10818, 2023)

## What this evaluates

Evaluates generative language models on a suite of multiple-choice and open-ended benchmarks covering reasoning, commonsense, multitask proficiency, and truthfulness. It measures accuracy across diverse domains to assess generalization and the impact of data combination strategies.

## Datasets

- **AI2 Reasoning Challenge (ARC)** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **TruthfulQA** — total ?; splits: test (-1)
- **BigBench** — total ?; splits: test (-1)
- **HumanEval** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted answers across all benchmark items. Computed as the number of matches between model predictions and gold labels divided by the total number of items.
- `RRGS` — range: [0, 1]
  - Risk of Random Guessing Score. Measures the variance of MMLU sub-item scores around the 25% random baseline using average L1 distance. Formula: RRGS = 1 - (1/N) * sum(|s_i - 0.25|) for i=1 to N.

## Input / output format

**Input**: Multiple-choice questions or open-ended prompts with few-shot examples (0, 5, 10, or 25 shots depending on benchmark) provided via the Eleuther AI LM Evaluation Harness.

**Output**: Model-generated text completion or selected option.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)

def compute_rrgs(sub_scores, baseline=0.25):
    n = len(sub_scores)
    avg_l1 = sum(abs(s - baseline) for s in sub_scores) / n
    return 1 - avg_l1
```

## Common pitfalls

- TruthfulQA is initialized with 0-shot in the harness but consistently runs as a 6-shot task.
- RRGS evaluates MMLU sub-item variance around 25% to detect random guessing, not standard task accuracy.
- Lower training loss does not directly correlate with higher benchmark accuracy.

## Evidence (verbatim from paper)

> Since 25% in MMLU represents the baseline score for a guess, this metric evaluates the variance using average l1 distance around this base value across all sub-items. A larger variance would suggest a reduced likelihood of predictions resulting from mere chance. Given a MMLU score vector X of length N with sub-item scores s1, s2, ..., sn, RRGS can be formulated as: RRGS = 1 - (1/N) * sum(|si - 0.25|)

## Citation

```bibtex
@misc{shen2023slimpajamadc,
  title={SlimPajama-DC: Understanding Data Combinations for LLM Training},
  author={Zhiqiang Shen et al. (2023)},
  year={2023},
  note={arXiv:2309.10818}
}
```

- arXiv: 2309.10818