Clone Detection Eval

---
name: clone-detection-eval
description: Evaluates a model's ability to determine whether two code snippets share the same semantics or to retrieve relevant code snippets from a repository. It probes semantic code similarity and code retrieval capabilities. Use when the user wants to benchmark on BigCloneBench, POJ-104, or asks about evaluating this task. Reports Overall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.04664
  bibtex_key: lu2021codexglue
  confidence: high
---

# clone-detection-eval

> CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation — Lu et al. (2021) (arXiv:2102.04664, 2021)

## What this evaluates

Evaluates a model's ability to determine whether two code snippets share the same semantics or to retrieve relevant code snippets from a repository. It probes semantic code similarity and code retrieval capabilities.

## Datasets

- **BigCloneBench** — total ?; splits: test (-1)
- **POJ-104** — total ?; splits: test (-1)

## Metrics

- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for binary classification of code pair similarity.
- `MAP` — range: [0, 1]
  - Mean Average Precision for retrieving 499 code snippets.
- `Overall` **(primary)** — range: [0, 1]
  - Average of F1 (BigCloneBench) and MAP (POJ-104) scores.

## Input / output format

**Input**: Pairs of code snippets (BigCloneBench) or a query code snippet with a candidate pool (POJ-104).

**Output**: Binary label (same/different semantics) or ranked list of retrieved code snippets.

## Scoring recipe

```python
For BigCloneBench: compute F1 on binary predictions.
For POJ-104: compute MAP over retrieved candidates.
Overall = (F1 + MAP) / 2
```

## Common pitfalls

- Models often ignore code structure (ASTs, data flow) which significantly impacts similarity measurement.
- POJ-104 retrieval task requires ranking 499 candidates, not just binary classification.

## Evidence (verbatim from paper)

> The task of the BigCloneBench dataset is formulated as a binary classification to predict whether a given pair of codes has the same semantics, with the F1 score used as the evaluation metric. The task of the POJ-104 dataset aims to retrieve 499 codes for a given code from the development/test set for validation/testing, with the Mean Average Precision (MAP) as the evaluation metric. The overall score of the clone detection task is the average value of F1 and MAP scores.

## Citation

```bibtex
@misc{lu2021codexglue,
  title={CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation},
  author={Lu et al. (2021)},
  year={2021},
  note={arXiv:2102.04664}
}
```

- arXiv: 2102.04664