Moe Routing Eval

---
name: moe-routing-eval
description: Evaluates the routing stability, robustness, and task performance of Sparse Mixture of Experts (SMoE) models enhanced with similarity or attention-aware mechanisms. It probes how token-level routing decisions affect language modeling, image classification, and downstream fine-tuning under both clean and perturbed conditions. Use when the user wants to benchmark on Wikitext-103, ImageNet-1K, ImageNet-C, ImageNet-A, ImageNet-R, ImageNet-O, SST5, SST2, Banking-77, or asks about evaluating this task. Reports perplexity, Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.00792
  bibtex_key: nguyen2025improvingrouting
  confidence: high
---

# moe-routing-eval

> Improving Routing in Sparse Mixture of Experts with Graph of Tokens — Tam Nguyen et al. (2025) (arXiv:2505.00792, 2025)

## What this evaluates

Evaluates the routing stability, robustness, and task performance of Sparse Mixture of Experts (SMoE) models enhanced with similarity or attention-aware mechanisms. It probes how token-level routing decisions affect language modeling, image classification, and downstream fine-tuning under both clean and perturbed conditions.

## Datasets

- **Wikitext-103** — total ?; splits: val (-1), test (-1)
- **ImageNet-1K** — total ?; splits: train (-1), val (-1), test (-1)
- **ImageNet-C** — total ?; splits: test (-1)
- **ImageNet-A** — total ?; splits: test (-1)
- **ImageNet-R** — total ?; splits: test (-1)
- **ImageNet-O** — total ?; splits: test (-1)
- **SST5** — total ?; splits: test (-1)
- **SST2** — total ?; splits: test (-1)
- **Banking-77** — total ?; splits: test (-1)

## Metrics

- `perplexity` **(primary)** — range: other
  - Exponential of the negative average log-likelihood of the ground-truth tokens: exp(-1/N * Σ log p(y_i | y_<i)). Lower values indicate better language modeling performance.
- `Top-1 accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly predicted class labels out of the total number of instances: sum(preds == gold) / N. Higher values indicate better classification or fine-tuning performance.
- `routing fluctuation rate` — range: percent
  - The percentage of tokens that change their assigned expert between consecutive training epochs (specifically epochs 59 and 60). Lower values indicate more stable routing.
- `decision entropy rate` — range: other
  - The average entropy of the probability distribution over experts for each token routing decision. Lower values indicate more confident and consistent expert selection.
- `load balancing score` — range: other
  - Measures the uniformity of token distribution across experts, typically evaluated via variance or standard deviation of expert assignment counts. More uniform distribution indicates better load balancing.

## Input / output format

**Input**: Tokenized text sequences for language modeling and fine-tuning tasks; image patches processed through a Vision MoE (V-MoE) architecture for classification tasks.

**Output**: For LM: probability distribution over the vocabulary for the next token. For classification/fine-tuning: predicted class label.

## Scoring recipe

```python
# Perplexity
ppl = exp(-mean(log(probs[range(len(tokens)), tokens])))
# Accuracy
acc = sum(preds == gold) / len(gold)
# Fluctuation
fluct = count(tokens_changed_expert_between_epoch_59_and_60) / total_tokens
# Entropy
entropy = mean(-sum(p_expert_given_token * log(p_expert_given_token)))
```

## Common pitfalls

- Confusing clean vs. attacked dataset results when comparing perplexity scores.
- Misinterpreting lower perplexity as worse performance (it is better).
- Overlooking that routing fluctuation and entropy are internal diagnostic metrics, not task performance metrics.
- Assuming the method replaces the base MoE architecture rather than acting as a plug-and-play enhancement to baselines like GLAM, X-MoE, and SMoE-dropout.

## Evidence (verbatim from paper)

> The models are evaluated on validation and test sets using perplexity scores (lower is better) on clean data and in adversarial scenarios, i.e. under word-swap attacks.

## Citation

```bibtex
@misc{nguyen2025improvingrouting,
  title={Improving Routing in Sparse Mixture of Experts with Graph of Tokens},
  author={Tam Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2505.00792}
}
```

- arXiv: 2505.00792