Magicvl 2b Eval

---
name: magicvl-2b-eval
description: Evaluates a lightweight vision-language model's performance on reasoning, OCR, and real-world understanding benchmarks, alongside deployment efficiency metrics like inference latency and throughput on mobile hardware. Use when the user wants to benchmark on HallusionBench, MMBench, RealworldQA, MMStar, OCRBench, AI2D, TextVQA, CRPE, MME Realworld, DocVQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.01540
  bibtex_key: liu2025magicvl2b
  confidence: high
---

# magicvl-2b-eval

> MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning — Yi Liu et al. (arXiv:2508.01540, 2025)

## What this evaluates

Evaluates a lightweight vision-language model's performance on reasoning, OCR, and real-world understanding benchmarks, alongside deployment efficiency metrics like inference latency and throughput on mobile hardware.

## Datasets

- **HallusionBench** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **RealworldQA** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **AI2D** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **CRPE** — total ?; splits: test (-1)
- **MME Realworld** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions or correctly classified instances across each benchmark.
- `ViT latency` — range: seconds
  - Wall-clock time in seconds to run the visual encoder inference on the target hardware.
- `LLM latency` — range: seconds
  - Wall-clock time in seconds to run the language model inference on the target hardware.
- `throughput` — range: tokens/s
  - Number of generated tokens per second during inference.

## Input / output format

**Input**: Image input paired with a text prompt/question.

**Output**: Text response generated by the model.

## Scoring recipe

```python
# For benchmark accuracy
 correct = 0
 for pred, gold in zip(predictions, gold_labels):
     if pred == gold:
         correct += 1
 accuracy = (correct / len(gold_labels)) * 100

# For deployment metrics
 viT_latency = time_end_vit - time_start_vit
 throughput = total_tokens_generated / total_inference_time
```

## Common pitfalls

- Comparing models across different parameter scales without normalization
- Dynamic resolution changes token count, which may affect latency/throughput comparisons if not accounted for
- On-device latency measurements depend heavily on hardware configuration and background processes

## Evidence (verbatim from paper)

> As shown in Table[3], MagicVL-2B reduces the total number of tokens by approximately 37.8% (0.52 M vs 0.81 M) during the evaluation, while also achieving improved accuracy (74.5% vs 74.3%).

## Citation

```bibtex
@misc{liu2025magicvl2b,
  title={MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning},
  author={Yi Liu et al.},
  year={2025},
  note={arXiv:2508.01540}
}
```

- arXiv: 2508.01540