Llava Scissor Eval

---
name: llava-scissor-eval
description: Evaluates the effectiveness of a training-free token compression method for video large language models across various video understanding tasks, including QA, long-video understanding, and multi-choice benchmarks, under different token retention ratios. Use when the user wants to benchmark on ActivityNet-QA, Video-ChatGPT, Next-QA, Egoschema, MLVU, Video-MME, VideoMMMU, MVBench, or asks about evaluating this task. Reports Avg.(%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.21862
  bibtex_key: sun2025llavascissor
  confidence: high
---

# llava-scissor-eval

> LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs — Sun et al. (2025) (arXiv:2506.21862, 2025)

## What this evaluates

Evaluates the effectiveness of a training-free token compression method for video large language models across various video understanding tasks, including QA, long-video understanding, and multi-choice benchmarks, under different token retention ratios.

## Datasets

- **ActivityNet-QA** — total ?; splits: test (-1)
- **Video-ChatGPT** — total ?; splits: test (-1)
- **Next-QA** — total ?; splits: test (-1)
- **Egoschema** — total ?; splits: test (-1)
- **MLVU** — total ?; splits: test (-1)
- **Video-MME** — total ?; splits: test (-1)
- **VideoMMMU** — total ?; splits: test (-1)
- **MVBench** — total ?; splits: test (-1)

## Metrics

- `Avg.(%)` **(primary)** — range: percent
  - Percentage of correct answers on each benchmark. The headline metric is the average accuracy across all evaluated benchmarks.

## Input / output format

**Input**: Video frames processed by a vision encoder (SIGLIP) and a text question/prompt.

**Output**: Text answer generated by the LLM (Qwen 2.5).

## Scoring recipe

```python
def compute_avg_accuracy(predictions, gold_answers):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if pred == gold)
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Token retention ratio is controlled by a similarity threshold $\tau$, so the exact number of retained tokens varies across datasets with different video distributions.
- Comparisons are only fair when using the same modified base model (LLaVA-OneVision with SIGLIP and Qwen 2.5), not the original LLaVA-OneVision.
- FLOPs are calculated only for the LLM stage (prefilling and decoding), excluding vision encoder and compression overhead.

## Evidence (verbatim from paper)

> | Method | Retention ratio | EgoSchema | MLVU | VideoMME | VideoMMMU | Avg.(%) |

## Citation

```bibtex
@misc{sun2025llavascissor,
  title={LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs},
  author={Sun et al. (2025)},
  year={2025},
  note={arXiv:2506.21862}
}
```

- arXiv: 2506.21862