Codebleu

---
name: codebleu
description: Evaluates the validity of the CodeBLEU metric for code synthesis by measuring its correlation with human programmer judgments across text-to-code generation, code translation, and code refinement tasks. Use when the user has predictions and gold and needs to compute CodeBLEU.
metadata:
  skill_kind: metric
  source_arxiv: 2009.10297
  bibtex_key: ren2020codebleu
  confidence: high
---

# codebleu

> CodeBLEU: a Method for Automatic Evaluation of Code Synthesis — Ren et al. (2020) (arXiv:2009.10297, 2020)

## What this evaluates

Evaluates the validity of the CodeBLEU metric for code synthesis by measuring its correlation with human programmer judgments across text-to-code generation, code translation, and code refinement tasks.

## Datasets

- **Iyer et al. [2018] dataset** — total 104000; splits: train (100000), val (2000), test (2000)
- **Code translation dataset (Java to C#)** — total 11800; splits: dev (500), test (1000)
- **Tufano et al. [2019] dataset** — total ?; splits: train_small (46680), dev_small (5835), test_small (5835), train_medium (52364), dev_medium (6545), test_medium (6545)

## Metrics

- `CodeBLEU` **(primary)** — range: [0, 100]
  - Combines weighted n-gram matching (BLEU), AST subtree alignment, and semantic similarity via data-flow analysis using hyperparameters α, β, γ, δ. Scaled to 0-100.

## Input / output format

**Input**: Natural language documentation and class environment (variable names/types, method names/return types) for text-to-code; Java method code for translation; buggy Java function code for refinement.

**Output**: Generated code (Java method/function or C# method).

## Scoring recipe

```python
def compute_codebleu(predictions, references):
    bleu = compute_ngram_precision(predictions, references)
    ast_match = compute_ast_subtree_match(predictions, references)
    df_match = compute_dataflow_similarity(predictions, references)
    # Default weights: alpha=0.25, beta=0.25, gamma=0.25, delta=0.25
    score = 0.25 * bleu + 0.25 * ast_match + 0.25 * df_match
    return score * 100  # Paper reports scores in [0, 100]
```

## Common pitfalls

- CodeBLEU scores are scaled to 0-100, not 0-1.
- Human evaluation uses a 1-5 Likert scale, not a percentage or 0-1 score.
- The metric's validity is assessed via Pearson correlation with human scores, not direct task accuracy.
- Hyperparameters α, β, γ, δ significantly impact correlation; default is not always optimal.

## Evidence (verbatim from paper)

> For each task, we calculate the Pearson correlation coefficient to check the correlation between the scores given by our proposed CodeBLEU and the scores assigned by programmers (human evaluation scores).

## Citation

```bibtex
@misc{ren2020codebleu,
  title={CodeBLEU: a Method for Automatic Evaluation of Code Synthesis},
  author={Ren et al. (2020)},
  year={2020},
  note={arXiv:2009.10297}
}
```

- arXiv: 2009.10297