Flashcache Eval

---
name: flashcache-eval
description: Evaluates the effectiveness of a frequency-domain-guided KV cache compression method (FlashCache) on multimodal long-context understanding tasks. It measures how well the model preserves accuracy under varying KV cache retention ratios and quantifies the computational overhead and decoding latency compared to baseline eviction methods. Use when the user wants to benchmark on MileBench, MUIRBench, MMMU, V*, HR-Bench, FAVOR-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.16786
  bibtex_key: yang2025flashcache
  confidence: high
---

# flashcache-eval

> Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach — Yang et al. (2025) (arXiv:2511.16786, 2025)

## What this evaluates

Evaluates the effectiveness of a frequency-domain-guided KV cache compression method (FlashCache) on multimodal long-context understanding tasks. It measures how well the model preserves accuracy under varying KV cache retention ratios and quantifies the computational overhead and decoding latency compared to baseline eviction methods.

## Datasets

- **MileBench** — total ?; splits: test (-1)
- **MUIRBench** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **V*** — total ?; splits: test (-1)
- **HR-Bench** — total ?; splits: test (-1)
- **FAVOR-Bench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions or tasks. Computed as (number of correct predictions / total number of instances) * 100.
- `decoding_latency` — range: ms
  - Time in milliseconds to generate a 100-token output sequence, measured across varying input lengths (2K to 64K tokens).
- `method_overhead` — range: ms
  - Additional computational time in milliseconds introduced by the KV cache eviction/compression method itself, measured for input lengths 2K, 4K, and 8K.

## Input / output format

**Input**: Multimodal prompts (images, videos, text) processed by MLLMs (LLaVA-OneVision-1.5-8B-Instruct, Qwen2.5-VL-7B/32B-Instruct) with KV cache retention ratio $\rho$ applied.

**Output**: Generated text responses evaluated against ground truth labels for accuracy.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- KV retention ratio $\rho$ is a critical hyperparameter; performance degrades sharply below 0.1, so results must be reported at specific ratios (0.2, 0.1, 0.05).
- Several baselines (H2O, LOOK-M, MEDA) encounter Out-Of-Memory (OOM) errors on long-context benchmarks, preventing direct comparison at lower retention ratios.
- HR-Bench accuracy is evaluated using a separate LLM judge (Qwen2-32B), which may introduce judge bias compared to exact-match metrics.

## Evidence (verbatim from paper)

> We employ Qwen2.5-VL-7B for the experiment with retention ratios of 0.1 and 0.05 and present the accuracy metrics for different compression methods.

## Citation

```bibtex
@misc{yang2025flashcache,
  title={Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2511.16786}
}
```

- arXiv: 2511.16786