Code Pretraining Impact Eval

---
name: code-pretraining-impact-eval
description: This evaluation protocol measures the impact of code data proportions and quality during LLM pre-training on downstream capabilities. It probes natural language reasoning, world knowledge, code generation, and generative text quality across different model initialization and pre-training mixture variants. Use when the user wants to benchmark on NL Reasoning Benchmarks, World Knowledge Tasks, Code Benchmarks (Python), Dolly-200-English, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.10914
  bibtex_key: arya2024tocodeornotcode
  confidence: medium
---

# code-pretraining-impact-eval

> To Code, or Not To Code? Exploring Impact of Code in Pre-training — Arya et al. (2024) (arXiv:2408.10914, 2024)

## What this evaluates

This evaluation protocol measures the impact of code data proportions and quality during LLM pre-training on downstream capabilities. It probes natural language reasoning, world knowledge, code generation, and generative text quality across different model initialization and pre-training mixture variants.

## Datasets

- **NL Reasoning Benchmarks** — total ?; splits: test (-1)
- **World Knowledge Tasks** — total ?; splits: test (-1)
- **Code Benchmarks (Python)** — total ?; splits: test (-1)
- **Dolly-200-English** — total ?; splits: test (-1)

## Metrics

- `pass@1` **(primary)** — range: percent
  - The fraction of generated code solutions that pass all provided unit tests on the first attempt. Reported as a percentage.
- `win-rate` — range: percent
  - Pairwise comparison of model generations against a baseline on open-ended prompts. Calculated as the percentage of wins minus losses relative to total comparisons.
- `relative improvement` — range: percent
  - Percentage change in metric score compared to a baseline model: ((new_score - baseline_score) / baseline_score) * 100.

## Input / output format

**Input**: Natural language questions, reasoning prompts, or coding tasks presented as text prompts.

**Output**: Generated text responses or Python code solutions.

## Scoring recipe

```python
def score_code(predictions, gold_tests):
    passed = 0
    for pred in predictions:
        if run_tests(pred, gold_tests): passed += 1
    return (passed / len(predictions)) * 100

def score_winrate(model_a, model_b, prompts):
    wins, losses, ties = 0, 0, 0
    for p in prompts:
        res = judge(model_a.generate(p), model_b.generate(p))
        if res == 'A': wins += 1
        elif res == 'B': losses += 1
        else: ties += 1
    return ((wins - losses) / (wins + losses + ties)) * 100
```

## Common pitfalls

- The paper reports relative improvements rather than absolute scores in the text, which can obscure baseline performance levels.
- Code evaluation is specifically measured in Python, so results may not generalize to other programming languages without re-evaluation.
- Win-rates are calculated against a text-only baseline, not an absolute standard, making cross-paper comparisons difficult.

## Evidence (verbatim from paper)

> In code evaluation, there is a linear increase in performance as the amount of code increases, with the best model being a code-only model. As observable in Figure [4] right inset, the 100% code leads to a 2.6x increase in the code benchmarks compared to the 25% code model. As expected, for the model with 0% code, the average pass@1 score drops to 0.

## Citation

```bibtex
@misc{arya2024tocodeornotcode,
  title={To Code, or Not To Code? Exploring Impact of Code in Pre-training},
  author={Arya et al. (2024)},
  year={2024},
  note={arXiv:2408.10914}
}
```

- arXiv: 2408.10914