Classeval Eval

---
name: classeval-eval
description: Evaluates large language models' ability to generate complete Python classes with interdependent methods, rather than standalone functions. It probes long-context code reasoning, dependency modeling, and the effectiveness of holistic versus incremental generation strategies. Use when the user wants to benchmark on ClassEval, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.01861
  bibtex_key: du2023classeval
  confidence: high
---

# classeval-eval

> ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation — Du et al. (2023) (arXiv:2308.01861, 2023)

## What this evaluates

Evaluates large language models' ability to generate complete Python classes with interdependent methods, rather than standalone functions. It probes long-context code reasoning, dependency modeling, and the effectiveness of holistic versus incremental generation strategies.

## Datasets

- **ClassEval** — total 100; splits: test (100); repo https://github.com/FudanSELab/ClassEval

## Metrics

- `pass@1` **(primary)** — range: [0, 1]
  - The proportion of generated classes that successfully pass all provided unit tests (both method-level and class-level) within a 5-second execution timeout per test case.

## Input / output format

**Input**: A structured class skeleton containing class-level information (class name, description, imports, constructor) and method-level information (method signature, functional description, parameter/return descriptions, and example I/O).

**Output**: A complete Python class implementation that conforms to the specified interface and passes the provided test suite.

## Scoring recipe

```python
def compute_pass_rate(predictions, test_suites):
    passed = 0
    for code, suite in zip(predictions, test_suites):
        try:
            if run_tests_with_timeout(code, suite, timeout=5.0):
                passed += 1
        except Exception:
            pass
    return passed / len(predictions)
```

## Common pitfalls

- Models often generate independent methods instead of a cohesive class, failing class-level tests that check inter-method dependencies.
- Method-level tests verify both return values and class field states, unlike standard function benchmarks that only check returns.
- The 5-second execution timeout may cause false negatives for models generating inefficient loops or blocking I/O operations.

## Evidence (verbatim from paper)

> Typically, LLMs generate code snippets based on input descriptions and the correctness is verified with the provided test suite.

## Citation

```bibtex
@misc{du2023classeval,
  title={ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation},
  author={Du et al. (2023)},
  year={2023},
  note={arXiv:2308.01861}
}
```

- arXiv: 2308.01861