Pevlm Longvideo Eval

---
name: pevlm-longvideo-eval
description: Evaluates the accuracy and latency-constrained performance of vision-language models on long-video understanding tasks. It measures how well models preserve temporal reasoning and answer questions about extended video sequences under strict computational and time budgets. Use when the user wants to benchmark on LongVideoBench, VideoMME, EgoSchema, MVBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.19651
  bibtex_key: kang2025pevlm
  confidence: high
---

# pevlm-longvideo-eval

> PEVLM: Parallel Encoding for Vision-Language Models — Kang et al. (2025) (arXiv:2506.19651, 2025)

## What this evaluates

Evaluates the accuracy and latency-constrained performance of vision-language models on long-video understanding tasks. It measures how well models preserve temporal reasoning and answer questions about extended video sequences under strict computational and time budgets.

## Datasets

- **LongVideoBench** — total ?; splits: test (-1)
- **VideoMME** — total ?; splits: test (-1)
- **EgoSchema** — total ?; splits: test (-1)
- **MVBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total dataset. In latency-constrained settings, samples exceeding the time limit are counted as failures.
- `latency-constrained accuracy` — range: percent
  - Accuracy computed on samples that complete inference within a specified time threshold (e.g., 20s, 30s, 40s). Samples exceeding the limit are treated as incorrect.
- `speedup` — range: other
  - Ratio of baseline inference time to PEVLM inference time, measured separately for the attention module and the full LLM.

## Input / output format

**Input**: Video frames encoded into tokens, concatenated with a system prompt and a question. Context blocks are fixed at ~4096 tokens, with a shared sink block derived from initial frames or the system prompt.

**Output**: Textual answers to video understanding questions.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip())
    return (correct / len(gold)) * 100

def compute_latency_constrained_accuracy(predictions, gold, latencies, threshold):
    correct = 0
    total = len(gold)
    for p, g, t in zip(predictions, gold, latencies):
        if t <= threshold and p.strip() == g.strip():
            correct += 1
    return (correct / total) * 100
```

## Common pitfalls

- Temperature for the APE baseline must be fixed at T=1.0; lowering it further skews attention distributions and degrades accuracy.
- Context block size is strictly fixed at 4096 tokens across all methods; smaller blocks reduce accuracy while larger blocks hurt latency.
- Latency-constrained accuracy counts samples exceeding the time limit as failures (incorrect), rather than excluding them from the evaluation denominator.

## Evidence (verbatim from paper)

> This experiment aims to evaluate the impact of PEVLM on accuracy in long-video understanding tasks. ... we introduce a latency threshold during inference: any sample that exceeds this latency limit is treated as a failure.

## Citation

```bibtex
@misc{kang2025pevlm,
  title={PEVLM: Parallel Encoding for Vision-Language Models},
  author={Kang et al. (2025)},
  year={2025},
  note={arXiv:2506.19651}
}
```

- arXiv: 2506.19651