Commit Message Completion Eval

---
name: commit-message-completion-eval
description: Evaluates how well models generate or complete commit messages given code diffs and optional historical context. It probes the model's ability to follow coding conventions, match ground truth exactly, and maintain semantic similarity under varying context lengths. Use when the user wants to benchmark on CMG_test, or asks about evaluating this task. Reports ExactMatch@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.07655
  bibtex_key: eliseeva2023commit
  confidence: high
---

# commit-message-completion-eval

> From Commit Message Generation to History-Aware Commit Message Completion — Eliseeva et al. (2023) (arXiv:2308.07655, 2023)

## What this evaluates

Evaluates how well models generate or complete commit messages given code diffs and optional historical context. It probes the model's ability to follow coding conventions, match ground truth exactly, and maintain semantic similarity under varying context lengths.

## Datasets

- **CMG_test** — total ?; splits: test (-1); repo https://github.com/JetBrains-Research/commit_message_generation

## Metrics

- `B-Norm` — range: [0, 1]
  - Normalized BLEU score variant measuring n-gram overlap between predicted and ground truth commit messages.
- `Edit Similarity` — range: [0, 1]
  - Ratio of matching characters or tokens after optimal alignment, measuring surface-level string similarity.
- `ExactMatch@1` **(primary)** — range: [0, 1]
  - Fraction of instances where the top-1 generated commit message exactly matches the ground truth.
- `ExactMatch@2` — range: [0, 1]
  - Fraction of instances where the ground truth appears in the top-2 generated candidates.

## Input / output format

**Input**: Code diff (and optionally previous commit messages for history) plus a user-typed commit message prefix (for completion) or empty prefix (for generation).

**Output**: A single commit message string.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em1 = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    em2 = sum(1 for p, g in zip(predictions, golds) if g in p or p in g) / len(golds)
    b_norm = compute_normalized_bleu(predictions, golds, n=4)
    ed_sim = compute_edit_similarity(predictions, golds)
    return {'B-Norm': b_norm, 'EdSim': ed_sim, 'EM@1': em1, 'EM@2': em2}
```

## Common pitfalls

- Evaluating on filtered subsets (e.g., First Sentence, Verb-Direct Object) inflates metrics and fails to reflect real-world commit diversity.
- Confusing the generation setting (0% context) with the completion setting (25% context), as metrics trend differently and completion is inherently easier.

## Evidence (verbatim from paper)

> From Table III, we observe that B-Norm and Edit Similarity metrics across all the models and settings increase when adding history.

## Citation

```bibtex
@misc{eliseeva2023commit,
  title={From Commit Message Generation to History-Aware Commit Message Completion},
  author={Eliseeva et al. (2023)},
  year={2023},
  note={arXiv:2308.07655}
}
```

- arXiv: 2308.07655