Codenet Classification Eval

---
name: codenet-classification-eval
description: Evaluates a model's ability to classify source code into the programming problem it was submitted to solve. It probes code representation learning and structural understanding by mapping code samples to their corresponding problem classes. Use when the user wants to benchmark on CodeNet (Java250, Python800, C++1000, C++1400), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2105.12655
  bibtex_key: puri2021codenet
  confidence: high
---

# codenet-classification-eval

> CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks — Puri et al. (2021) (arXiv:2105.12655, 2021)

## What this evaluates

Evaluates a model's ability to classify source code into the programming problem it was submitted to solve. It probes code representation learning and structural understanding by mapping code samples to their corresponding problem classes.

## Datasets

- **CodeNet (Java250, Python800, C++1000, C++1400)** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/IBM/Project_CodeNet

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified code samples out of the total test samples. Calculated as (correct predictions / total test instances) * 100.

## Input / output format

**Input**: Source code snippet (represented as token sequence, bag-of-tokens, or graph structure) corresponding to a programming problem submission.

**Output**: Predicted problem class label (integer index or string name of the coding problem).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- C++ benchmarks contain noise from copied constants and data structures across different problem submissions, which can artificially lower model performance.
- Tokenization strategy heavily impacts results; using a language-specific tokenizer (e.g., Clang for C) vs. generic tokenizers changes the feature space significantly.
- The 80/20 split is random, which may cause data leakage if similar code submissions are split across train and test sets.

## Evidence (verbatim from paper)

> In each experiment, 20% of the code samples are used for testing, while the rest are split in 4:1 for training and validation, respectively. Table 7 summarizes the classification accuracy for all models on all benchmarks.

## Citation

```bibtex
@misc{puri2021codenet,
  title={CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks},
  author={Puri et al. (2021)},
  year={2021},
  note={arXiv:2105.12655}
}
```

- arXiv: 2105.12655