Nl Code Pair Mining Eval

---
name: nl-code-pair-mining-eval
description: Evaluates a machine learning model's ability to automatically extract high-quality, aligned natural language intent and code snippet pairs from Stack Overflow posts. It probes the system's ranking capability, precision, and recall across different programming languages and feature combinations. Use when the user wants to benchmark on Stack Overflow NL-Code Pairs, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1805.08949
  bibtex_key: yin2018learning
  confidence: high
---

# nl-code-pair-mining-eval

> Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow — Pengcheng Yin et al. (arXiv:1805.08949, 2018)

## What this evaluates

Evaluates a machine learning model's ability to automatically extract high-quality, aligned natural language intent and code snippet pairs from Stack Overflow posts. It probes the system's ranking capability, precision, and recall across different programming languages and feature combinations.

## Datasets

- **Stack Overflow NL-Code Pairs** — total ?; splits: train (71828), val (7981), annotated (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Curve (AUC) computed over Precision-Recall and ROC curves. PR plots precision vs recall for top-k predictions; ROC plots true positive rate vs false positive rate.
- `Precision-Recall (PR)` — range: [0, 1]
  - Curve plotting precision against recall for top-k ranked predictions, where k ranges from 1 to the total number of candidates.
- `ROC` — range: [0, 1]
  - Receiver Operating Characteristic curve plotting true positive rate against false positive rate for the ranked candidate list.

## Input / output format

**Input**: Stack Overflow question title (intent) and its answers containing code blocks.

**Output**: Ranked list of candidate intent/snippet pairs with probability scores.

## Scoring recipe

```python
# For each SO question:
# 1. Extract candidate intent/snippet pairs from answers
# 2. Score and rank candidates by model probability
# 3. Compare ranked list against gold-standard annotations
# 4. Compute precision and recall at each rank k
# 5. Plot PR and ROC curves across all questions
# 6. Calculate AUC for both curves as summary metric
```

## Common pitfalls

- Correspondence features trained on full code blocks bias towards imports and assignments, hurting fine-grained snippet extraction.
- Structural features over-prioritize full code blocks or last lines, missing partial implementations.
- Cross-language transfer requires feature normalization due to different value ranges across languages.
- Gold annotations may be incomplete, leading to false negatives in evaluation.

## Evidence (verbatim from paper)

> We present the results using standard precision-recall (PR) and Receiver Operating Characteristic (ROC) curves. In short, a PR curve shows the precision w.r.t. recall for the top- k predictions in the ranked list, with k from 1 to the number of candidates. A ROC curve plots the true positive rates w.r.t. false positive rates in similar fashion. We also compute the Area Under the Curve (AUC) scores for all ROC curves.

## Citation

```bibtex
@misc{yin2018learning,
  title={Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow},
  author={Pengcheng Yin et al.},
  year={2018},
  note={arXiv:1805.08949}
}
```

- arXiv: 1805.08949