Scicode Eval

---
name: scicode-eval
description: Probes large language models' ability to perform scientific reasoning, domain-specific knowledge recall, and code synthesis on real-world research problems. It evaluates performance on both decomposed subproblems and full main problems under varying conditions of background knowledge and context carry-over. Use when the user wants to benchmark on SciCode, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.13168
  bibtex_key: tian2024scicode
  confidence: high
---

# scicode-eval

> SciCode: A Research Coding Benchmark Curated by Scientists — Tian et al. (2024) (arXiv:2407.13168, 2024)

## What this evaluates

Probes large language models' ability to perform scientific reasoning, domain-specific knowledge recall, and code synthesis on real-world research problems. It evaluates performance on both decomposed subproblems and full main problems under varying conditions of background knowledge and context carry-over.

## Datasets

- **SciCode** — total 80; splits: test (80)

## Metrics

- `pass@1` **(primary)** — range: percent
  - pass@1 = (number of problems solved correctly on the first attempt / total number of problems) × 100. It measures the percentage of problems where the model's initial generated code passes all validation test cases.

## Input / output format

**Input**: Problem description (main problem or subproblem), optional gold scientific background text, and optionally generated or gold solutions from previous subproblems.

**Output**: Executable code snippet solving the specified problem.

## Scoring recipe

```python
correct = 0
for problem in dataset:
    code = model.generate(problem.prompt)
    if execute_and_pass_tests(code, problem.test_cases):
        correct += 1
return (correct / len(dataset)) * 100
```

## Common pitfalls

- Confusing the standard setup (no background knowledge, no carry-over) with the background-knowledge setup, which yields significantly higher scores.
- Assuming high subproblem pass@1 rates translate to main problem success; the gap is large due to cumulative error and instruction-following demands.
- Ignoring the requirement to carry over generated solutions from previous subproblems in the standard setup, which tests context management and instruction following.

## Evidence (verbatim from paper)

> Table 2 presents results under the standard setup.444Without background and carrying over generated subproblem solutions. See §[2.4] for a more detailed discussion. For the easier subproblem-level evaluation, the state-of-the-art models we test solve 14%-26% of the subproblems. Among them, Claude3.5-Sonnet achieves the best performance, with a 26.0% pass@1 rate.

## Citation

```bibtex
@misc{tian2024scicode,
  title={SciCode: A Research Coding Benchmark Curated by Scientists},
  author={Tian et al. (2024)},
  year={2024},
  note={arXiv:2407.13168}
}
```

- arXiv: 2407.13168