Skillrouter Eval

---
name: skillrouter-eval
description: This benchmark evaluates the ability of LLM-based skill routers to accurately select the most relevant tool or function from a large-scale pool of candidate skills based on a user query. It probes retrieval and reranking capabilities under varying difficulty tiers and tests whether models can handle single-skill versus multi-skill routing scenarios. Use when the user wants to benchmark on SkillRouter Benchmark, or asks about evaluating this task. Reports Hit@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.22455
  bibtex_key: zheng2026skillrouter
  confidence: high
---

# skillrouter-eval

> SkillRouter: Retrieve-and-Rerank Skill Selection for LLM Agents at Scale — YanZhao Zheng et al. (2026) (arXiv:2603.22455, 2026)

## What this evaluates

This benchmark evaluates the ability of LLM-based skill routers to accurately select the most relevant tool or function from a large-scale pool of candidate skills based on a user query. It probes retrieval and reranking capabilities under varying difficulty tiers and tests whether models can handle single-skill versus multi-skill routing scenarios.

## Datasets

- **SkillRouter Benchmark** — total 75; splits: test (75)

## Metrics

- `Hit@1` **(primary)** — range: [0, 1]
  - Primary top-1 routing metric. Counts as success if any required skill is ranked first. For multi-skill queries, success is recorded when at least one ground-truth skill appears at rank 1. [0, 1]
- `MRR@10` — range: [0, 1]
  - Mean Reciprocal Rank at 10. Computes the average of 1/rank for the first relevant skill in the top-10 results. [0, 1]
- `nDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at 10. Measures ranking quality by discounting the relevance of correctly retrieved skills by their position, normalized by the ideal DCG. [0, 1]
- `Recall@10` — range: [0, 1]
  - Fraction of ground-truth skills retrieved within the top-10 candidates. Computed as |predicted[:10] ∩ gold| / |gold|. [0, 1]
- `Recall@20` — range: [0, 1]
  - Fraction of ground-truth skills retrieved within the top-20 candidates. Computed as |predicted[:20] ∩ gold| / |gold|. [0, 1]
- `Recall@50` — range: [0, 1]
  - Fraction of ground-truth skills retrieved within the top-50 candidates. Computed as |predicted[:50] ∩ gold| / |gold|. [0, 1]
- `FC@10` — range: [0, 1]
  - Full Coverage at 10. Indicates whether all required skills for a multi-skill query are retrieved within the top-10 candidates. [0, 1]

## Input / output format

**Input**: A user query paired with a list of candidate skills. Each skill is represented by three text fields: name, description, and body. For retrievers, the full skill pool (~80K) is used; for rerankers, the top-20 candidates from the retriever are provided.

**Output**: A ranked list of candidate skills (or a single top-1 skill selection for LLM-as-judge baselines).

## Scoring recipe

```python
def compute_metrics(predictions, gold_skills):
    # predictions: list of skill IDs ranked by model
    # gold_skills: set of required skill IDs for the query
    hit1 = 1.0 if any(s in gold_skills for s in predictions[:1]) else 0.0
    rr = [1.0/(i+1) for i, s in enumerate(predictions) if s in gold_skills]
    mrr = rr[0] if rr else 0.0
    recall10 = len(set(predictions[:10]) & gold_skills) / len(gold_skills)
    recall20 = len(set(predictions[:20]) & gold_skills) / len(gold_skills)
    recall50 = len(set(predictions[:50]) & gold_skills) / len(gold_skills)
    fc10 = 1.0 if gold_skills.issubset(set(predictions[:10])) else 0.0
    return hit1, mrr, recall10, recall20, recall50, fc10
```

## Common pitfalls

- Ignoring the single-skill vs. multi-skill distinction, which causes Hit@1 and coverage metrics (FC@10) to diverge significantly.
- Using only name and description (nd) inputs for rerankers actively degrades performance below encoder-only retrieval due to insufficient context.
- Assuming zero-shot general-purpose encoders outperform domain-adapted fine-tuned models without empirical verification on skill routing data.

## Evidence (verbatim from paper)

> We report seven metrics: Hit@1 (our primary top-1 routing metric), MRR@10 (mean reciprocal rank), nDCG@10 (normalized discounted cumulative gain), Recall@10, Recall@20, Recall@50, and FC@10. For multi-skill queries, Hit@1 counts success when any required skill is ranked first, while Recall@10 and FC@10 provide complementary coverage views.

## Citation

```bibtex
@misc{zheng2026skillrouter,
  title={SkillRouter: Retrieve-and-Rerank Skill Selection for LLM Agents at Scale},
  author={YanZhao Zheng et al. (2026)},
  year={2026},
  note={arXiv:2603.22455}
}
```

- arXiv: 2603.22455