Skillret Eval

---
name: skillret-eval
description: Evaluates the ability of embedding models and rerankers to accurately retrieve relevant software skills from a large, noisy library based on long, scenario-rich user queries. It probes ranking quality, recall, and completeness in a two-stage retrieve-then-rerank pipeline, highlighting the need for domain-specific fine-tuning over general semantic matching. Use when the user wants to benchmark on SkillRet, or asks about evaluating this task. Reports NDCG@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2605.05726
  bibtex_key: cho2026skillret
  confidence: high
---

# skillret-eval

> SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents — Cho et al. (2026) (arXiv:2605.05726, 2026)

## What this evaluates

Evaluates the ability of embedding models and rerankers to accurately retrieve relevant software skills from a large, noisy library based on long, scenario-rich user queries. It probes ranking quality, recall, and completeness in a two-stage retrieve-then-rerank pipeline, highlighting the need for domain-specific fine-tuning over general semantic matching.

## Datasets

- **SkillRet** — total 17810; splits: train (127190), test (2319); repo https://github.com/ThakiCloud/SKILLRET

## Metrics

- `NDCG@k` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank k. Measures the quality of the ranked list of retrieved skills, with higher scores indicating better alignment of ground-truth skills at the top of the list.
- `Recall@k` — range: [0, 1]
  - The fraction of ground-truth skills for a query that appear within the top-k retrieved results.
- `Completeness@k` — range: [0, 1]
  - The fraction of queries where all ground-truth skills are successfully retrieved within the top-k results (i.e., Recall@k = 1).

## Input / output format

**Input**: User query (typically long, scenario-rich text) and a candidate pool of skill documents (each containing name, description, and full Markdown body).

**Output**: A ranked list of top-k skill candidates returned by the retriever and/or reranker.

## Scoring recipe

```python
def compute_metrics(predictions, gold, k_values=[5, 10, 15]):
    metrics = {}
    for k in k_values:
        top_k = predictions[:k]
        recall = len(set(top_k) & set(gold)) / len(gold)
        completeness = 1.0 if recall == 1.0 else 0.0
        dcg = sum(1.0 / math.log2(i + 2) for i, s in enumerate(top_k) if s in gold)
        idcg = sum(1.0 / math.log2(i + 2) for i in range(len(gold)))
        ndcg = dcg / idcg if idcg > 0 else 0.0
        metrics[f'Recall@{k}'] = recall
        metrics[f'Completeness@{k}'] = completeness
        metrics[f'NDCG@{k}'] = ndcg
    return metrics
```

## Common pitfalls

- Using general-purpose rerankers can degrade performance due to domain mismatch, overriding correct results from specialized retrievers.
- High scores on general benchmarks like MTEB do not predict performance on skill retrieval, as the task requires identifying sparse capability signals in long, noisy queries.
- Models may distribute attention diffusely across entire queries instead of focusing on the few sentences that directly signal skill intent.

## Evidence (verbatim from paper)

> We report three metrics at $k\in{5,10,15}$: NDCG@$k$ measures ranking quality, Recall@$k$ measures the fraction of ground-truth skills retrieved, and Completeness@$k$ measures the fraction of queries where all ground-truth skills are retrieved, i.e., Recall@$k\=1$.

## Citation

```bibtex
@misc{cho2026skillret,
  title={SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents},
  author={Cho et al. (2026)},
  year={2026},
  note={arXiv:2605.05726}
}
```

- arXiv: 2605.05726