Flashvlm Eval

---
name: flashvlm-eval
description: Evaluates the robustness and efficiency of text-guided visual token pruning in large multimodal models. It probes whether aggressive token compression (retaining 32–128 tokens for images, 114–455 for video) degrades performance on image and video question-answering tasks, and measures cross-modal grounding quality via spatial alignment and semantic overlap metrics. Use when the user wants to benchmark on VQAv2, GQA, VizWiz, ScienceQA-IMG, TextVQA, POPE, MME, MMBench, MMBench-CN, MM Vet, TGIF-QA, MSVDQA, MSRVTT-QA, ActivityNet-QA, or asks about evaluating this task. Reports accuracy, average_accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.20561
  bibtex_key: cai2025flashvlm
  confidence: high
---

# flashvlm-eval

> FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models — Cai et al. (2025) (arXiv:2512.20561, 2025)

## What this evaluates

Evaluates the robustness and efficiency of text-guided visual token pruning in large multimodal models. It probes whether aggressive token compression (retaining 32–128 tokens for images, 114–455 for video) degrades performance on image and video question-answering tasks, and measures cross-modal grounding quality via spatial alignment and semantic overlap metrics.

## Datasets

- **VQAv2** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **VizWiz** — total ?; splits: test (-1)
- **ScienceQA-IMG** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **MMBench-CN** — total ?; splits: test (-1)
- **MM Vet** — total ?; splits: test (-1)
- **TGIF-QA** — total ?; splits: test (-1)
- **MSVDQA** — total ?; splits: test (-1)
- **MSRVTT-QA** — total ?; splits: test (-1)
- **ActivityNet-QA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions across benchmarks, computed via official evaluation scripts.
- `average_accuracy` **(primary)** — range: percent
  - Mean accuracy across all 10 image benchmarks or 3 video benchmarks, used as the headline performance indicator.
- `attention_distance` — range: other
  - Quantifies the spatial deviation of the model's final focus region from the ground truth, capturing RoPE-induced proximity bias.
- `score_map_entropy` — range: other
  - Measures the concentration of the token selection score distribution; lower values indicate less redundancy and more stable salient regions.
- `token_box_iou` — range: other
  - Intersection over Union between selected visual tokens and manually annotated ground-truth region boxes, evaluating semantic grounding precision.

## Input / output format

**Input**: Image or video frames (8 frames at 224px resolution for video) paired with a text prompt/question.

**Output**: Text response generated by the VLM; evaluated via exact match/accuracy for QA datasets or LLM-assistant scoring for open-ended benchmarks.

## Scoring recipe

```python
def compute_metrics(predictions, gold_answers, token_scores=None, gt_boxes=None):
    acc = sum(1 for p, g in zip(predictions, gold_answers) if is_correct(p, g)) / len(predictions)
    metrics = {'accuracy': acc}
    if token_scores is not None:
        metrics['score_map_entropy'] = -sum(s * log(s) for s in token_scores)
    if gt_boxes is not None:
        metrics['token_box_iou'] = compute_iou(selected_tokens, gt_boxes)
        metrics['attention_distance'] = compute_spatial_deviation(selected_tokens, gt_boxes)
    return metrics
```

## Common pitfalls

- Baseline performance numbers are directly copied from the VisPruner paper rather than re-run under identical configurations, which may introduce unfair comparison artifacts.
- Token budgets are reported as absolute counts (e.g., 128, 64, 32 for images; 455, 227, 114 for video) rather than fixed pruning ratios, making cross-model compression comparisons sensitive to base token counts.
- The 'beyond-lossless' accuracy gains (>100% relative to upper bound) stem from filtering noisy tokens rather than actual model improvement, which can mislead readers expecting monotonic degradation with pruning.

## Evidence (verbatim from paper)

> FlashVLM achieves an average accuracy of 100.60%, surpassing even the reported performance of the unpruned (upper-bound) model.

## Citation

```bibtex
@misc{cai2025flashvlm,
  title={FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2512.20561}
}
```

- arXiv: 2512.20561