Cubert Fine Tuning Eval

---
name: cubert-fine-tuning-eval
description: Evaluates contextual code embeddings on six Python source-code understanding tasks, including classification of variable misuse, incorrect binary operators, swapped operands, function-docstring mismatches, and exception types, plus a joint localization and repair task. Use when the user wants to benchmark on ETH Py150 Open Benchmarks, or asks about evaluating this task. Reports classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2001.00059
  bibtex_key: kanade2019learning
  confidence: high
---

# cubert-fine-tuning-eval

> Learning and Evaluating Contextual Embedding of Source Code — Kanade et al. (2019) (arXiv:2001.00059, 2019)

## What this evaluates

Evaluates contextual code embeddings on six Python source-code understanding tasks, including classification of variable misuse, incorrect binary operators, swapped operands, function-docstring mismatches, and exception types, plus a joint localization and repair task.

## Datasets

- **ETH Py150 Open Benchmarks** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `classification accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels or pointers out of total instances. For classification tasks, it is standard top-1 accuracy. For the localization/repair task, it requires exact match of both the localization pointer and the repair variable.

## Input / output format

**Input**: Tokenized Python function (or function-docstring pair) using a custom subword vocabulary. Inputs are padded or truncated to a fixed maximum length.

**Output**: For classification: a single class label. For localization/repair: two pointers (indices) indicating the misuse location and the correct variable to substitute.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Datasets are synthetically generated by replacing variables/operators/docstrings, so they may not reflect real-world bug distributions.
- Validation sets are heavily subsampled (e.g., down to 8,192) compared to test sets, which may cause evaluation variance.
- Tasks use the same base corpus (ETH Py150 Open) but with synthetic noise, so performance may not generalize to natural code.

## Evidence (verbatim from paper)

> To evaluate CuBERT, we design five classification tasks and a multi-headed pointer task. These are motivated by prior work, but unfortunately, the associated datasets come from different languages and varied sources. We want the tasks to be on Python code, and for accurate results, we ensure that there is no overlap between pre-training and fine-tuning datasets.

## Citation

```bibtex
@misc{kanade2019learning,
  title={Learning and Evaluating Contextual Embedding of Source Code},
  author={Kanade et al. (2019)},
  year={2019},
  note={arXiv:2001.00059}
}
```

- arXiv: 2001.00059