Hermes Video Understanding Eval

---
name: hermes-video-understanding-eval
description: Evaluates a training-free KV cache management framework for real-time streaming and offline video understanding. It probes the model's ability to maintain temporal coherence and answer questions accurately under strict token/memory budgets, while measuring inference efficiency. Use when the user wants to benchmark on StreamingBench, OVO-Bench, RVS (Ego & Movie), MVBench, VideoMME, Egoschema, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.14724
  bibtex_key: zhang2026hermes
  confidence: high
---

# hermes-video-understanding-eval

> HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding — Haowei Zhang et al. (arXiv:2601.14724, 2026)

## What this evaluates

Evaluates a training-free KV cache management framework for real-time streaming and offline video understanding. It probes the model's ability to maintain temporal coherence and answer questions accurately under strict token/memory budgets, while measuring inference efficiency.

## Datasets

- **StreamingBench** — total ?; splits: test (-1)
- **OVO-Bench** — total ?; splits: test (-1)
- **RVS (Ego & Movie)** — total ?; splits: test (-1)
- **MVBench** — total ?; splits: test (-1)
- **VideoMME** — total ?; splits: test (-1)
- **Egoschema** — total ?; splits: dev (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 100] percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.
- `score` — range: [1, 5] scale
  - 1-5 scale rating for open-ended answers, evaluated by GPT-3.5-turbo-0125 based on answer accuracy.
- `TTFT` — range: ms
  - Time to First Token, measured in milliseconds from query input to first output token generation.
- `TPOT` — range: ms
  - Time Per Output Token, measured in milliseconds per generated token.
- `peak_gpu_memory` — range: GB
  - Maximum GPU memory usage during inference, measured in gigabytes.

## Input / output format

**Input**: Video frames processed sequentially in chunks of 16 frames, fed into the backbone LLM. Text questions provided as prompts.

**Output**: Text responses, either multiple-choice selections or open-ended descriptions.

## Scoring recipe

```python
def compute_metrics(predictions, golds, llm_judge='gpt-3.5-turbo-0125'):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
    accuracy = (correct / len(golds)) * 100
    scores = []
    for p, g in zip(predictions, golds):
        score = llm_judge.evaluate(p, g, scale=(1, 5))
        scores.append(score)
    avg_score = sum(scores) / len(scores)
    return {'accuracy': accuracy, 'avg_score': avg_score}
```

## Common pitfalls

- Open-ended evaluation on RVS relies on GPT-3.5-turbo-0125 as a judge rather than exact string matching, which introduces LLM-judge bias and prompt sensitivity.
- Efficiency metrics (TTFT, TPOT, memory) are reported on a single A800 GPU with FP16 precision; results may vary significantly on consumer or different enterprise GPUs.
- The memory budget is fixed at 4K tokens for main results, but ablations show streaming tasks tolerate smaller budgets while long offline tasks degrade sharply below 4K.

## Evidence (verbatim from paper)

> On RVS-Ego and RVS-Movie (Tab. 2), we evaluate the model answer by GPT-3.5-turbo-0125 on accuracy and score (1–5 scale), consistent with compared baselines.

## Citation

```bibtex
@misc{zhang2026hermes,
  title={HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding},
  author={Haowei Zhang et al.},
  year={2026},
  note={arXiv:2601.14724}
}
```

- arXiv: 2601.14724