Ds 1000 Eval

---
name: ds-1000-eval
description: This benchmark evaluates a model's ability to generate correct, executable Python code for data science tasks, specifically focusing on NumPy operations. It probes functional correctness under natural language descriptions and tests robustness against surface-form and semantic perturbations of original StackOverflow problems. Use when the user wants to benchmark on numpy-100, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.11501
  bibtex_key: lai2022ds1000
  confidence: high
---

# ds-1000-eval

> DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation — Lai et al. (2022) (arXiv:2211.11501, 2022)

## What this evaluates

This benchmark evaluates a model's ability to generate correct, executable Python code for data science tasks, specifically focusing on NumPy operations. It probes functional correctness under natural language descriptions and tests robustness against surface-form and semantic perturbations of original StackOverflow problems.

## Datasets

- **numpy-100** — total 100; splits: test (100)

## Metrics

- `pass@1` **(primary)** — range: [0, 1]
  - Fraction of problems where at least one of the generated samples passes all provided test cases. Computed over sampled generations per problem.

## Input / output format

**Input**: Natural language problem description followed by a code template containing an `[insert]` placeholder, e.g., `Problem: <description>\n<code>\nimport numpy as np\n[insert]\nprint(result)\n</code>`

**Output**: Python code snippet that fills the `[insert]` placeholder to solve the problem.

## Scoring recipe

```python
def compute_pass_at_1(generated_codes, test_cases):
    passed_count = 0
    for code in generated_codes:
        try:
            exec(code, {"np": __import__("numpy")})
            if all(check_test_case(code, tc) for tc in test_cases):
                passed_count += 1
                break
        except Exception:
            pass
    return passed_count / len(generated_codes)
```

## Common pitfalls

- Models may memorize original StackOverflow solutions; perturbations are required to measure true generalization.
- Execution-based evaluation requires a secure sandbox; models might attempt to read/write files or access the network, which should be blocked.
- Surface-form constraints (e.g., specific API usage) are often overlooked if only functional test cases are checked.

## Evidence (verbatim from paper)

> At last, we equipped each problem and its perturbation with one test case and an automatic evaluation. Then we tested the performance of Codex-002 on them. We sampled 20 problems from numpy-100 and generated 10 samples for each problem with temperature set to 0.7, and top-p cutoff set to 0.95.

## Citation

```bibtex
@misc{lai2022ds1000,
  title={DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation},
  author={Lai et al. (2022)},
  year={2022},
  note={arXiv:2211.11501}
}
```

- arXiv: 2211.11501