Helm Lite Eval

---
name: helm-lite-eval
description: This evaluation probes the robustness of open benchmarks against test-set memorization and data leakage. It measures whether small language models can artificially inflate leaderboard scores by overfitting directly to public test sets, revealing flaws in current benchmarking practices. Use when the user wants to benchmark on HELM-lite, or asks about evaluating this task. Reports Exact Match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.00460
  bibtex_key: hasan2025pitfallsevaluating
  confidence: high
---

# helm-lite-eval

> Pitfalls of Evaluating Language Models with Open Benchmarks — Hasan et al. (2025) (arXiv:2507.00460, 2025)

## What this evaluates

This evaluation probes the robustness of open benchmarks against test-set memorization and data leakage. It measures whether small language models can artificially inflate leaderboard scores by overfitting directly to public test sets, revealing flaws in current benchmarking practices.

## Datasets

- **HELM-lite** — total ?; splits: test (-1)

## Metrics

- `Exact Match` **(primary)** — range: [0, 1]
  - 1 if the predicted answer exactly matches the gold answer (after normalization), else 0. Averaged over the test set.
- `Quasi Exact Match` — range: [0, 1]
  - Matches the gold answer after normalizing whitespace and case, as defined in the HELM benchmark.
- `F1 Score` — range: [0, 1]
  - Token-level harmonic mean of precision and recall for open-ended QA tasks.
- `BLEU-4` — range: [0, 100]
  - 4-gram BLEU score for machine translation evaluation.
- `Equivalent (CoT)` — range: [0, 1]
  - Checks if the predicted answer is mathematically equivalent to the gold answer, often leveraging chain-of-thought reasoning.

## Input / output format

**Input**: Text prompts from HELM-lite scenarios, including multiple-choice questions, open-ended QA, translation pairs, and math problems.

**Output**: Model-generated answers or translations, formatted according to each scenario's specific requirements.

## Scoring recipe

```python
def score(predictions, golds, metric):
    if metric in ["Exact Match", "Quasi Exact Match"]:
        return sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g)) / len(golds)
    elif metric == "F1 Score":
        return token_f1(predictions, golds)
    elif metric == "BLEU-4":
        return bleu_4(predictions, golds)
    elif metric == "Equivalent (CoT)":
        return math_equiv(predictions, golds)
    return 0.0
```

## Common pitfalls

- Training models on public test sets (data leakage) artificially inflates scores without reflecting true generalization.
- Different scenarios use different evaluation metrics, making direct cross-benchmark comparison difficult.

## Evidence (verbatim from paper)

> Each scenario is evaluated using metrics defined in the original HELM benchmark. MMLUExact MatchClaude 3.5 Sonnet80.9088.5091.6096.2094.8094.24

## Citation

```bibtex
@misc{hasan2025pitfallsevaluating,
  title={Pitfalls of Evaluating Language Models with Open Benchmarks},
  author={Hasan et al. (2025)},
  year={2025},
  note={arXiv:2507.00460}
}
```

- arXiv: 2507.00460