Coir Eval

---
name: coir-eval
description: Evaluates code information retrieval models across diverse tasks including text-to-code, code-to-code, code-to-text, and hybrid code retrieval. It probes a model's ability to handle semi-structured, syntactically complex code snippets and natural language queries across multiple programming languages and domains. Use when the user wants to benchmark on APPS, CosQA, Synthetic Text2SQL, CodeSearchNet, CodeSearchNet-CCR, CodeTransOcean-DL, CodeTransOcean-Contest, StackOverflow QA, CodeFeedQA, CodeFeedback-MT, or asks about evaluating this task. Reports nDCG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.02883
  bibtex_key: li2024coir
  confidence: high
---

# coir-eval

> CoIR: A Comprehensive Benchmark for Code Information Retrieval Models — Xiangyang Li et al. (2024) (arXiv:2407.02883, 2024)

## What this evaluates

Evaluates code information retrieval models across diverse tasks including text-to-code, code-to-code, code-to-text, and hybrid code retrieval. It probes a model's ability to handle semi-structured, syntactically complex code snippets and natural language queries across multiple programming languages and domains.

## Datasets

- **APPS** — total 14000; splits: train (5000), test (3800)
- **CosQA** — total 20600; splits: test (-1)
- **Synthetic Text2SQL** — total 106000; splits: train (100000), test (6000)
- **CodeSearchNet** — total 1000000; splits: train (905000), dev (41000), test (53000)
- **CodeSearchNet-CCR** — total 1000000; splits: train (905000), dev (41000), test (53000)
- **CodeTransOcean-DL** — total ?; splits: test (-1)
- **CodeTransOcean-Contest** — total 1000; splits: train (561), dev (226), test (446)
- **StackOverflow QA** — total 20000; splits: train (13000), dev (3000), test (2000)
- **CodeFeedQA** — total ?; splits: test (-1)
- **CodeFeedback-MT** — total 66000; splits: train (53000), test (13000)

## Metrics

- `nDCG` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at a given cutoff (typically @10 or @100). Computed as DCG@k divided by IDCG@k, where DCG ranks relevant documents by their graded relevance scores discounted logarithmically by position.
- `precision` — range: [0, 1]
  - Fraction of retrieved documents that are relevant at a specific cutoff k.
- `recall` — range: [0, 1]
  - Fraction of all relevant documents that are successfully retrieved at cutoff k.
- `MAP` — range: [0, 1]
  - Mean Average Precision across all queries, averaging the precision values at each rank where a relevant document is retrieved.

## Input / output format

**Input**: A query (natural language, code snippet, or mixed text/code) and a candidate corpus of code/text documents to retrieve from.

**Output**: A ranked list of corpus documents or relevance scores for each query, typically output as JSON.

## Scoring recipe

```python
def compute_ndcg(relevant_docs, predicted_ranking, k=10):
    dcg = 0.0
    for i, doc_id in enumerate(predicted_ranking[:k]):
        rel = 1 if doc_id in relevant_docs else 0
        dcg += rel / math.log2(i + 2)
    idcg = sum(1 / math.log2(i + 2) for i in range(min(len(relevant_docs), k)))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Models often overfit to specific benchmarks like CodeSearchNet, leading to inflated performance but poor generalization across diverse code domains.
- Multi-turn retrieval tasks require handling dialogue contexts exceeding 4,000 tokens, which exceeds the standard 512-token context window of many retrieval models.
- Code is semi-structured and syntactically complex, making standard text-based retrieval metrics insufficient without careful handling of code-specific tokenization and formatting.

## Evidence (verbatim from paper)

> Unlike traditional evaluations that require manual coding and result collection, CoIR offers an automated pipeline for both open-source and proprietary models, supporting metrics such as nDCG, precision, recall, and MAP. Results are stored in JSON format for easy access.

## Citation

```bibtex
@misc{li2024coir,
  title={CoIR: A Comprehensive Benchmark for Code Information Retrieval Models},
  author={Xiangyang Li et al. (2024)},
  year={2024},
  note={arXiv:2407.02883}
}
```

- arXiv: 2407.02883