Disrouter Routing Eval

---
name: disrouter-routing-eval
description: Evaluates a distributed multi-agent routing system's ability to autonomously assign queries to the most appropriate LLM based on intrinsic self-assessment. It probes the trade-off between routing accuracy and inference cost across diverse mathematical, commonsense, and reading comprehension benchmarks. Use when the user wants to benchmark on GSM8K, ARC, MMLU, RACE_HIGH, OpenbookQA, DROP, CosmosQA, SQuAD, HellaSwag, HeadQA, or asks about evaluating this task. Reports utility.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.19208
  bibtex_key: zheng2025disrouter
  confidence: high
---

# disrouter-routing-eval

> DiSRouter: Distributed Self-Routing for LLM Selections — Zheng et al. (2025) (arXiv:2510.19208, 2025)

## What this evaluates

Evaluates a distributed multi-agent routing system's ability to autonomously assign queries to the most appropriate LLM based on intrinsic self-assessment. It probes the trade-off between routing accuracy and inference cost across diverse mathematical, commonsense, and reading comprehension benchmarks.

## Datasets

- **GSM8K** — total ?; splits: test (-1)
- **ARC** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **RACE_HIGH** — total ?; splits: test (-1)
- **OpenbookQA** — total ?; splits: test (-1)
- **DROP** — total ?; splits: test (-1)
- **CosmosQA** — total ?; splits: test (-1)
- **SQuAD** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **HeadQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` — range: [0, 1]
  - Standard exact-match or option-matching accuracy computed per dataset. Averaged across datasets for final reporting.
- `cost` — range: [0, 1]
  - Average normalized inference cost of the routed models per query. Each model is assigned a fixed normalized cost between 0 and 1 based on its size.
- `utility` **(primary)** — range: [0, 1]
  - A tunable weighted metric balancing accuracy and cost, controlled by a global preference factor α. Higher α prioritizes cost reduction, while lower α prioritizes accuracy.

## Input / output format

**Input**: Natural language query or task prompt from the benchmark dataset, evaluated in a zero-shot Chain-of-Thought manner.

**Output**: Routing decision specifying which LLM from the pool handles the query, followed by the LLM's generated response or a rejection signal.

## Scoring recipe

```python
def compute_metrics(predictions, routing_decisions, golds, model_costs, alpha):
    acc = sum(1 for p, g in zip(predictions, golds) if is_correct(p, g)) / len(golds)
    cost = sum(model_costs[m] for m in routing_decisions) / len(golds)
    utility = alpha * acc + (1 - alpha) * (1 - cost)
    return acc, cost, utility
```

## Common pitfalls

- Uses validation sets as test sets for benchmarks with hidden test splits (e.g., GSM8K, MMLU), which may inflate or deflate reported accuracy compared to official leaderboards.
- Cost is a normalized fixed value per model rather than actual latency or token usage, making cross-system cost comparisons sensitive to the normalization scheme.
- The utility metric is highly sensitive to the global preference factor α; results reported for one α value do not generalize to other scenarios without re-evaluation.

## Evidence (verbatim from paper)

> We evaluate LLMs in a zero-shot Chain-of-Thought (CoT, Wei et al., [2022]) manner and use accuracy as the evaluation metric for all these datasets. To evaluate the overall routing performance, we utilize the utility metric introduced in §[2.1], Equation ([2]), which can be adjusted for different scenarios by setting a corresponding value for α.

## Citation

```bibtex
@misc{zheng2025disrouter,
  title={DiSRouter: Distributed Self-Routing for LLM Selections},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2510.19208}
}
```

- arXiv: 2510.19208