Opencompass Downstream Eval

---
name: opencompass-downstream-eval
description: Evaluates language model performance across five diverse downstream benchmarks spanning commonsense reasoning, science QA, and complex reasoning. It specifically probes how dynamic expert routing mechanisms adapt to input difficulty compared to fixed Top-K routing. Use when the user wants to benchmark on PIQA, Hellaswag, ARC-e, CommonsenseQA, BBH, or asks about evaluating this task. Reports accuracy (score).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.07652
  bibtex_key: huang2024harder
  confidence: high
---

# opencompass-downstream-eval

> Harder Tasks Need More Experts: Dynamic Routing in MoE Models — Quzhe Huang et al. (arXiv:2403.07652, 2024)

## What this evaluates

Evaluates language model performance across five diverse downstream benchmarks spanning commonsense reasoning, science QA, and complex reasoning. It specifically probes how dynamic expert routing mechanisms adapt to input difficulty compared to fixed Top-K routing.

## Datasets

- **PIQA** — total ?; splits: test (-1)
- **Hellaswag** — total ?; splits: test (-1)
- **ARC-e** — total ?; splits: test (-1)
- **CommonsenseQA** — total ?; splits: test (-1)
- **BBH** — total ?; splits: test (-1)

## Metrics

- `accuracy (score)` **(primary)** — range: percent
  - Percentage of correctly predicted answers or multiple-choice selections across all benchmark tasks, averaged across datasets.

## Input / output format

**Input**: Multiple-choice or open-ended prompts for each downstream benchmark task.

**Output**: Model's predicted answer or selected choice.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Scores are averaged across heterogeneous benchmarks (PIQA, Hellaswag, ARC-e, CommonsenseQA, BBH), which can mask task-specific degradation.
- The dynamic routing threshold p is tuned at inference time on a pre-trained model rather than during fine-tuning, which may not reflect optimal training-time routing behavior.
- Different evaluation harnesses (e.g., OpenCompass vs. lm-evaluation-harness) may yield slightly different accuracy scores due to prompt formatting or answer extraction differences.

## Evidence (verbatim from paper)

> We use opencompass to evaluate our model. Table 1 shows the performance of different models on downstream tasks. Overall, the MoE models outperform the Dense models. Among all the MoE variants, our proposed Dynamic Adaptive MoE demonstrates the best performance, achieving at least a 0.7% higher score on average compared to other models.

## Citation

```bibtex
@misc{huang2024harder,
  title={Harder Tasks Need More Experts: Dynamic Routing in MoE Models},
  author={Quzhe Huang et al.},
  year={2024},
  note={arXiv:2403.07652}
}
```

- arXiv: 2403.07652