Mobileaibench Eval

---
name: mobileaibench-eval
description: Evaluates the task performance, latency, and hardware utilization of quantized LLMs and LMMs on real mobile devices across standard NLP, multi-modal, and trust & safety benchmarks. Use when the user wants to benchmark on Databricks, HotpotQA, sql-create-context, CNN, XSum, VQA-v2, GQA, VisWiz, TextVQA, SQA, AlpacaEval, MT-Bench, MMLU, GSM8K, TruthQA, BBQ, SC-101, Adv-Inst, DNA, Priv-Lk, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.10290
  bibtex_key: murthy2024mobileaibench
  confidence: high
---

# mobileaibench-eval

> MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases — Murthy et al. (2024) (arXiv:2406.10290, 2024)

## What this evaluates

Evaluates the task performance, latency, and hardware utilization of quantized LLMs and LMMs on real mobile devices across standard NLP, multi-modal, and trust & safety benchmarks.

## Datasets

- **Databricks** — total ?; splits: test (-1)
- **HotpotQA** — total ?; splits: test (-1)
- **sql-create-context** — total ?; splits: test (-1)
- **CNN** — total ?; splits: test (-1)
- **XSum** — total ?; splits: test (-1)
- **VQA-v2** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **VisWiz** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **SQA** — total ?; splits: test (-1)
- **AlpacaEval** — total ?; splits: test (-1)
- **MT-Bench** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **TruthQA** — total ?; splits: test (-1)
- **BBQ** — total ?; splits: test (-1)
- **SC-101** — total ?; splits: test (-1)
- **Adv-Inst** — total ?; splits: test (-1)
- **DNA** — total ?; splits: test (-1)
- **Priv-Lk** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered instances across VQA, MMLU, GSM8K, and Trust & Safety tasks.
- `Exact Match (EM)` — range: [0, 1]
  - Percentage of predictions that exactly match the ground truth answer.
- `F1 Score (F1)` — range: [0, 1]
  - Harmonic mean of precision and recall based on token overlap between prediction and gold.
- `Rouge-1 (R1)` — range: [0, 1]
  - Unigram overlap F-score between generated summary and reference.
- `Rouge-L (RL)` — range: [0, 1]
  - Longest common subsequence F-score between generated summary and reference.
- `Win-Rate` — range: [0, 1]
  - Proportion of times the model's response is preferred over a baseline in pairwise comparisons (AlpacaEval).
- `MT-Bench Score` — range: [0, 10]
  - Average score assigned by an LLM-as-a-judge across multi-turn conversation prompts.

## Input / output format

**Input**: Text prompts for NLP tasks; image and text prompts for VQA tasks; instruction or multi-turn prompts for trust & safety and benchmark tasks.

**Output**: Model-generated text responses.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type in ['qa', 'vqa', 'trust_safety', 'mmlu', 'gsm8k']:
        return sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    elif task_type == 'summarization':
        return rouge1(predictions, golds), rougel(predictions, golds)
    elif task_type == 'text2sql':
        return sql_parser_score(predictions, golds), levenshtein_score(predictions, golds)
    elif task_type == 'alpaca':
        return pairwise_win_rate(predictions, golds)
    elif task_type == 'mtbench':
        return llm_judge_score(predictions, golds)
```

## Common pitfalls

- Quantization at 3-bit causes severe performance drops for most LMMs, while some models (e.g., Moondream2) remain robust.
- Desktop evaluations may not capture real-world mobile constraints like thermal throttling, RAM limits, or actual latency (time-to-first-token).
- Different models and tasks exhibit varying sensitivity to quantization, so reporting only average scores can hide critical degradation patterns.

## Evidence (verbatim from paper)

> For question answering tasks, we employ Exact Match (EM) and F1 Score (F1). In the context of Text-to-SQL tasks, we utilize the SQL Parser (SP) and Levenshtein Score (LS). For summarization tasks, we measure performance using Rouge-1 (R1) and Rouge-L (RL). Additionally, we use Win-Rate for AlpacaEval, Score for MT Bench, and Accuracy for both MMLU and GSM8K.

## Citation

```bibtex
@misc{murthy2024mobileaibench,
  title={MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases},
  author={Murthy et al. (2024)},
  year={2024},
  note={arXiv:2406.10290}
}
```

- arXiv: 2406.10290