Concode Eval

---
name: concode-eval
description: Probes a model's ability to generate syntactically valid Java member functions from natural language documentation, conditioned on a full class environment including variable types, method signatures, and their interdependencies. It evaluates context-aware code generation, identifier disambiguation, and code reusability. Use when the user wants to benchmark on CONCODE, or asks about evaluating this task. Reports Exact match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1808.09588
  bibtex_key: iyer2018mapping
  confidence: high
---

# concode-eval

> Mapping Language to Code in Programmatic Context — Iyer et al. (2018) (arXiv:1808.09588, 2018)

## What this evaluates

Probes a model's ability to generate syntactically valid Java member functions from natural language documentation, conditioned on a full class environment including variable types, method signatures, and their interdependencies. It evaluates context-aware code generation, identifier disambiguation, and code reusability.

## Datasets

- **CONCODE** — total 100000; splits: train (-1), val (-1), test (2000); repo https://github.com/sriniyer/concode

## Metrics

- `Exact match accuracy` **(primary)** — range: [0, 1]
  - 1 if the generated code exactly matches the reference code, else 0. Averaged over the dataset.
- `BLEU` — range: [0, 1] or percent
  - An n-gram precision-based metric that measures the overlap between predicted and reference code tokens to provide partial credit for syntactically similar outputs.

## Input / output format

**Input**: Natural language documentation combined with programmatic context (variable types, method signatures, and their interdependencies), restricted to ≤200 tokens total.

**Output**: Java member function source code (generated via syntactically valid production rules), restricted to ≤150 tokens or ≤500 production rules.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    exact_matches = sum(1 for p, r in zip(predictions, references) if p == r)
    exact_acc = exact_matches / len(references)
    bleu = compute_bleu(references, predictions)  # standard n-gram precision
    return exact_acc, bleu
```

## Common pitfalls

- Exact match is extremely strict and penalizes minor formatting or whitespace differences, often underestimating functional correctness.
- BLEU score focuses on n-gram overlap and may assign high scores to syntactically valid but semantically incorrect code.
- Evaluation requires strict tokenization (camel-case splitting, lower-casing) which significantly impacts metric scores if not replicated.

## Evidence (verbatim from paper)

> To evaluate the quality of the output, we use Exact match accuracy between the reference and generated code. As a measure of partial credit, we also compute the BLEU score (Papineni et al., 2002), following recent work on code generation (Ling et al., 2016; Yin and Neubig, 2017). BLEU is an n-gram precision-based metric that will be higher when more subparts of the predicted code match the provided reference.

## Citation

```bibtex
@misc{iyer2018mapping,
  title={Mapping Language to Code in Programmatic Context},
  author={Iyer et al. (2018)},
  year={2018},
  note={arXiv:1808.09588}
}
```

- arXiv: 1808.09588