Ltdr Eval

---
name: ltdr-eval
description: Evaluates the vision-language understanding and domain generalization capabilities of Mixture-of-Experts (MoE) models. It tests how well a long-tailed distribution-aware router preserves routing variance for vision tokens while maintaining load balancing for language tokens, impacting both accuracy and inference efficiency. Use when the user wants to benchmark on GQA, ScienceQA-IMG, TextVQA, POPE, MME, MMBench, MM-Vet, PACS, VLCS, Office-Home, DomainNet, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.01351
  bibtex_key: cai2025ltldr
  confidence: high
---

# ltdr-eval

> Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model — Cai et al. (2025) (arXiv:2507.01351, 2025)

## What this evaluates

Evaluates the vision-language understanding and domain generalization capabilities of Mixture-of-Experts (MoE) models. It tests how well a long-tailed distribution-aware router preserves routing variance for vision tokens while maintaining load balancing for language tokens, impacting both accuracy and inference efficiency.

## Datasets

- **GQA** — total ?; splits: test (-1)
- **ScienceQA-IMG** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **MM-Vet** — total ?; splits: test (-1)
- **PACS** — total ?; splits: test (-1)
- **VLCS** — total ?; splits: test (-1)
- **Office-Home** — total ?; splits: test (-1)
- **DomainNet** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers or class labels out of the total number of instances.
- `MME score` — range: other
  - Raw aggregate score from the MME benchmark, combining perception and cognition sub-tasks.
- `Avg accuracy` — range: percent
  - Arithmetic mean of accuracy scores across all benchmarks except MME.
- `training time (s)` — range: other
  - Total wall-clock time in seconds required to complete training.
- `inference time (s)` — range: other
  - Total wall-clock time in seconds required to process all benchmark instances.

## Input / output format

**Input**: For vision-language tasks: an image paired with a text question/prompt. For vision-only tasks: a single image.

**Output**: For vision-language tasks: a generated text answer. For vision-only tasks: a predicted class label.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if normalize(p) == normalize(g))
    return (correct / len(gold_labels)) * 100

def compute_avg_accuracy(accuracies):
    valid_accs = [a for a in accuracies if a != 'MME_score']
    return sum(valid_accs) / len(valid_accs)
```

## Common pitfalls

- MME reports a raw score (e.g., 1363.5) rather than a percentage, so it should not be averaged with other accuracy metrics.
- The 'Avg' column explicitly excludes the MME benchmark from the average calculation.
- Results marked with an asterisk (∗) indicate potential overlap with the model's training data, which may inflate reported accuracy.

## Evidence (verbatim from paper)

> As shown in Tab.[1], we evaluate the performance of our method on three image question-answering benchmarks and four benchmark toolkits, reporting both accuracy and the parameter of LLMs. ... We calculate the average performance “Avg” across all datasets except for MME.

## Citation

```bibtex
@misc{cai2025ltldr,
  title={Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2507.01351}
}
```

- arXiv: 2507.01351