Lime Mmt47 Eval

---
name: lime-mmt47-eval
description: Evaluates the ability of lightweight Mixture of Experts (MoE) parameter-efficient fine-tuning methods to generalize across diverse multimodal tasks. It probes how well shared PEFT modules with expert modulation vectors capture task-specific specialization without learned routing parameters. Use when the user wants to benchmark on MMT-47, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.02338
  bibtex_key: kowsher2026lime
  confidence: high
---

# lime-mmt47-eval

> LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning — Kowsher et al. (2026) (arXiv:2604.02338, 2026)

## What this evaluates

Evaluates the ability of lightweight Mixture of Experts (MoE) parameter-efficient fine-tuning methods to generalize across diverse multimodal tasks. It probes how well shared PEFT modules with expert modulation vectors capture task-specific specialization without learned routing parameters.

## Datasets

- **MMT-47** — total 158000; splits: train (158000), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels out of total test instances. Reported as mean ± standard deviation across 5 random seeds.
- `throughput` — range: other
  - Training speed measured in samples processed per second on a single H100 GPU.
- `trainable_parameters` — range: other
  - Total count of trainable parameters in the model.

## Input / output format

**Input**: Multimodal instances (text, image, or video) paired with ground-truth labels from the MMT-47 benchmark mixture.

**Output**: Predicted class labels or regression values for each instance.

## Scoring recipe

```python
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = (correct / len(gold_labels)) * 100
mean_acc = np.mean([accuracy(seed) for seed in 5_seeds])
std_acc = np.std([accuracy(seed) for seed in 5_seeds])
```

## Common pitfalls

- Performance is highly sensitive to the number of experts (E); accuracy peaks at E=4-5 and degrades with more experts due to insufficient data per expert.
- The routing threshold θ significantly impacts results; too low (θ→0) activates noisy experts, while too high (θ→1) discards useful secondary experts, with optimal around 0.7.
- Load balancing coefficients must be carefully tuned; zero balancing causes expert collapse, while excessive balancing (>1.0) suppresses natural specialization and hurts accuracy.

## Evidence (verbatim from paper)

> Figure 3(c-d) reports GLUE accuracy as we vary the number of experts from 1 to 10. Both LiME and MoELoRA peak at 3-5 experts (stars), suggesting that a moderate number of experts often provides the best balance between added capacity and how well experts can be trained.

## Citation

```bibtex
@misc{kowsher2026lime,
  title={LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning},
  author={Kowsher et al. (2026)},
  year={2026},
  note={arXiv:2604.02338}
}
```

- arXiv: 2604.02338