Mobile Mmlu Eval

---
name: mobile-mmlu-eval
description: Evaluates language models' understanding of mobile-specific domains and tasks under on-device constraints. It probes the models' ability to answer multiple-choice questions across 80 real-world mobile domains, emphasizing practical usability, privacy, and personalization in daily mobile interactions. Use when the user wants to benchmark on Mobile-MMLU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.20786
  bibtex_key: [REDACTED]
  confidence: high
---

# mobile-mmlu-eval

> Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark — Bsharat et al. (2025) (arXiv:2503.20786, 2025)

## What this evaluates

Evaluates language models' understanding of mobile-specific domains and tasks under on-device constraints. It probes the models' ability to answer multiple-choice questions across 80 real-world mobile domains, emphasizing practical usability, privacy, and personalization in daily mobile interactions.

## Datasets

- **Mobile-MMLU** — total 16186; splits: test (16186); repo https://github.com/VILA-Lab/Mobile-MMLU

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: A multiple-choice question with four options (A, B, C, D) covering a specific mobile domain.

**Output**: The model must output exactly one of the four option letters (A, B, C, or D) corresponding to the correct answer.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Answer order bias: Small LLMs are highly sensitive to the position of the correct answer (A/B/C/D), causing performance variance exceeding 10% if options are not systematically balanced or randomized.
- Model size does not guarantee performance: Smaller models (e.g., 3B parameters) can outperform larger ones (e.g., 8B parameters) on mobile-specific tasks due to architectural or training differences.
- Zero-shot evaluation: The benchmark is evaluated in a zero-shot setting without in-context examples, which may yield different results compared to few-shot practices common in other benchmarks.

## Evidence (verbatim from paper)

> Given that Mobile-MMLU and Mobile-MMLU-Pro consist entirely of multiple-choice questions, we focus on accuracy as our primary evaluation metric. This approach allows for objective comparison across models while maintaining consistency with existing benchmarking practices in the field.

## Citation

```bibtex
@misc{bsharat2025mobilemmlu,
  title={Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark},
  author={Bsharat et al. (2025)},
  year={2025},
  note={arXiv:2503.20786}
}
```

- arXiv: 2503.20786