Kernelbench Eval

---
name: kernelbench-eval
description: Evaluates large language models' ability to generate functionally correct and hardware-efficient CUDA kernels for PyTorch workloads. It probes the models' capacity for low-level systems programming, hardware-aware optimization, and debugging execution/functional errors under one-shot prompting. Use when the user wants to benchmark on KernelBench, or asks about evaluating this task. Reports fast_p.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.10517
  bibtex_key: ouyang2025kernelbench
  confidence: high
---

# kernelbench-eval

> KernelBench: Can LLMs Write Efficient GPU Kernels? — Ouyang et al. (2025) (arXiv:2502.10517, 2025)

## What this evaluates

Evaluates large language models' ability to generate functionally correct and hardware-efficient CUDA kernels for PyTorch workloads. It probes the models' capacity for low-level systems programming, hardware-aware optimization, and debugging execution/functional errors under one-shot prompting.

## Datasets

- **KernelBench** — total ?; splits: test (-1)

## Metrics

- `fast_p` **(primary)** — range: percent
  - Percentage of problems where the model-generated kernel is functionally correct and achieves at least a p-times speedup over the PyTorch Eager baseline. At p=1, it measures the percentage of kernels faster than PyTorch Eager.

## Input / output format

**Input**: A PyTorch model/task to optimize, accompanied by one in-context example of a simple add operator kernel showing the expected input/output format.

**Output**: A CUDA kernel implementation (ModelNew) written as source code.

## Scoring recipe

```python
def compute_fast_p(predictions, baselines, p=1.0):
    correct_count = 0
    for pred in predictions:
        if is_functionally_correct(pred, baselines['gold']):
            pred_time = profile_on_gpu(pred, device='L40S')
            base_time = baselines['pytorch_eager_time']
            if pred_time <= base_time / p:
                correct_count += 1
    return (correct_count / len(predictions)) * 100
```

## Common pitfalls

- Confusing fast_p with a continuous speedup ratio; it is a thresholded percentage metric that only counts kernels meeting both correctness and speed thresholds.
- Assuming hardware portability; kernels optimized for one GPU (e.g., L40S) often show significant speedup variations on different architectures (e.g., A10G).
- Overlooking that fast_0 counts all functionally correct kernels regardless of speed, while fast_1 strictly requires >1x speedup over the baseline.

## Evidence (verbatim from paper)

> We profile the generated code on an NVIDIA L40S GPU, and measure the $	ext{fast}_{p}$ metric across all problems. Figure 4 shows the distribution of $	ext{fast}_{p}$ as $p$ varies, indicating the percentage of kernels that are $p$-times faster than the PyTorch Eager baseline (the top right of the plot is better).

## Citation

```bibtex
@misc{ouyang2025kernelbench,
  title={KernelBench: Can LLMs Write Efficient GPU Kernels?},
  author={Ouyang et al. (2025)},
  year={2025},
  note={arXiv:2502.10517}
}
```

- arXiv: 2502.10517