Swimba Standard Bench Eval

---
name: swimba-standard-bench-eval
description: Evaluates language understanding, reasoning, and knowledge recall capabilities of a Switch Mamba model on standard multiple-choice benchmarks. It also measures inference efficiency (throughput, latency, FLOPs) to assess the computational trade-offs of the parameter-space mixture-of-experts design. Use when the user wants to benchmark on BoolQ, OpenBookQA, RTE, MMLU, PIQA, WinoGrande, HellaSwag, ARC-Challenge, ARC-Easy, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.06938
  bibtex_key: du2026swimba
  confidence: high
---

# swimba-standard-bench-eval

> Swimba: Switch Mamba Model Scales State Space Models — Du et al. (2026) (arXiv:2603.06938, 2026)

## What this evaluates

Evaluates language understanding, reasoning, and knowledge recall capabilities of a Switch Mamba model on standard multiple-choice benchmarks. It also measures inference efficiency (throughput, latency, FLOPs) to assess the computational trade-offs of the parameter-space mixture-of-experts design.

## Datasets

- **BoolQ** — total ?; splits: test (-1)
- **OpenBookQA** — total ?; splits: test (-1)
- **RTE** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **PIQA** — total ?; splits: test (-1)
- **WinoGrande** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **ARC-Challenge** — total ?; splits: test (-1)
- **ARC-Easy** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - The fraction of examples where the correct choice has the highest total log-likelihood.
- `Normalized accuracy` — range: [0, 1]
  - The fraction of examples where the correct choice has the highest length-normalized log-likelihood, computed by dividing each choice log-likelihood by its answer length in bytes.

## Input / output format

**Input**: Multiple-choice questions or prompts with candidate answers, processed via LM-Evaluation-Harness.

**Output**: Log-likelihood scores for each candidate answer choice.

## Scoring recipe

```python
def compute_accuracy(lls, gold_idx):
    return int(np.argmax(lls) == gold_idx)

def compute_normalized_accuracy(lls, lengths, gold_idx):
    norm_lls = [ll / length for ll, length in zip(lls, lengths)]
    return int(np.argmax(norm_lls) == gold_idx)

# Aggregate over dataset
acc_scores = [compute_accuracy(pred['lls'], gold) for pred, gold in zip(predictions, golds)]
norm_acc_scores = [compute_normalized_accuracy(pred['lls'], pred['lengths'], gold) for pred, gold in zip(predictions, golds)]
return sum(acc_scores) / len(golds), sum(norm_acc_scores) / len(golds)
```

## Common pitfalls

- Using greedy decoding or generation-based accuracy instead of log-likelihood evaluation, which LM-Evaluation-Harness requires for multiple-choice tasks.
- Ignoring length normalization when computing normalized accuracy, leading to inflated scores for longer answer choices.
- Reporting single-run results without standard deviations, whereas the paper explicitly reports subscripts for standard deviation across runs.

## Evidence (verbatim from paper)

> For performance with LM-Evaluation-Harness, we report accuracy, the fraction of examples where the correct choice has the highest total log-likelihood; and normalized accuracy, the fraction of examples where the correct choice has the highest length-normalized log-likelihood, computed by dividing each choice log-likelihood by its answer length in bytes

## Citation

```bibtex
@misc{du2026swimba,
  title={Swimba: Switch Mamba Model Scales State Space Models},
  author={Du et al. (2026)},
  year={2026},
  note={arXiv:2603.06938}
}
```

- arXiv: 2603.06938