Refinedweb Zero Shot Eval

---
name: refinedweb-zero-shot-eval
description: Evaluates the zero-shot generalization capability of autoregressive language models across multiple task aggregates. It measures how well models trained on raw web data perform on downstream tasks without any fine-tuning or prompt engineering. Use when the user wants to benchmark on Eleuther AI LM evaluation harness (zero-shot aggregates), or asks about evaluating this task. Reports zero-shot accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.01116
  bibtex_key: [REDACTED]
  confidence: high
---

# refinedweb-zero-shot-eval

> The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only — Penedo et al. (2023) (arXiv:2306.01116, 2023)

## What this evaluates

Evaluates the zero-shot generalization capability of autoregressive language models across multiple task aggregates. It measures how well models trained on raw web data perform on downstream tasks without any fine-tuning or prompt engineering.

## Datasets

- **Eleuther AI LM evaluation harness (zero-shot aggregates)** — total ?; splits: test (-1); repo https://github.com/EleutherAI/lm-evaluation-harness

## Metrics

- `zero-shot accuracy` **(primary)** — range: percent
  - Percentage of tasks where the model's generated completion exactly matches the ground-truth label or correct answer in a zero-shot setting. Averaged across task aggregates (small, core, main, ext).

## Input / output format

**Input**: Task description or prompt followed by the question, provided in a zero-shot format without in-context examples.

**Output**: Autoregressive text completion. Accuracy is determined by parsing the generated text and comparing it to the gold label.

## Scoring recipe

```python
def compute_zero_shot_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Perplexity or validation loss alone can be misleading and does not correlate well with end-task zero-shot performance.
- Comparing models trained/evaluated with different codebases, numerical precisions, or prompts introduces externalities that skew results.
- Filtering heuristics do not guarantee consistent improvements across different source datasets; they often require source-specific tuning.

## Evidence (verbatim from paper)

> We base our evaluation on the popular Eleuther AI evaluation harness (Gao et al., 2021), allowing us to evaluate across a wide range of tasks in the zero-shot setting. ... Average accuracy in zero-shot on the small-agg aggregate.

## Citation

```bibtex
@misc{penedo2023refinedweb,
  title={The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only},
  author={Penedo et al. (2023)},
  year={2023},
  note={arXiv:2306.01116}
}
```

- arXiv: 2306.01116