Leum Vl Eval

---
name: leum-vl-eval
description: Evaluates a video-language model's capacity for structured video understanding across six professional dimensions (subject, aesthetics, camera language, editing, narrative, dissemination) while maintaining general multimodal capabilities. It probes timeline-grounded reasoning, temporal localization, and document/OCR comprehension. Use when the user wants to benchmark on FeedBench, Open Benchmarks (Video-MME, MVBench, MMBench-EN, etc.), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.20354
  bibtex_key: he2026leumvl
  confidence: high
---

# leum-vl-eval

> Leum-VL Technical Report — He et al. (2026) (arXiv:2603.20354, 2026)

## What this evaluates

Evaluates a video-language model's capacity for structured video understanding across six professional dimensions (subject, aesthetics, camera language, editing, narrative, dissemination) while maintaining general multimodal capabilities. It probes timeline-grounded reasoning, temporal localization, and document/OCR comprehension.

## Datasets

- **FeedBench** — total ?; splits: test (-1); repo https://github.com/leum-team/leum-v1
- **Open Benchmarks (Video-MME, MVBench, MMBench-EN, etc.)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard benchmark accuracy calculated as the percentage of correctly predicted answers or dimension scores. For FeedBench, an LLM judge (DeepSeek-V3) evaluates structured outputs against expert-sensitive label boundaries.

## Input / output format

**Input**: Video content sampled at 4 FPS with a maximum of 768 frames, accompanied by a text prompt/question. Total input is capped at 50K tokens per video.

**Output**: Text response answering the prompt or providing structured labels for the video dimensions.

## Scoring recipe

```python
def compute_accuracy(predictions, gold, use_llm_judge=False):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if use_llm_judge:
            verdict = llm_judge_evaluate(pred, gold_item)  # DeepSeek-V3
            correct += verdict
        else:
            correct += (pred == gold_item)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- FeedBench relies on DeepSeek-V3 as an LLM judge rather than deterministic ground truth, which may introduce evaluation variance or bias.
- Fixed frame sampling (FPS=4, max 768 frames) and token limits may truncate fast-paced or long videos, affecting temporal grounding metrics.
- Cross-benchmark comparisons mix official reported values with local reproductions (marked †), complicating direct performance baselines.

## Evidence (verbatim from paper)

> On RefineShot, we improve overall accuracy by +4.2 pp (56.9 vs. 52.7), with the largest gains on Shot Framing (+13.4pp), Camera Angle (+10.8pp), and Lighting Type (+6.4pp) dimensions that correspond directly to the SV6D schema. Eval: FPS=4, max 768 frames, max 50K tokens/video. Judge: DeepSeek-V3 (DeepSeek-AI, 2025).

## Citation

```bibtex
@misc{he2026leumvl,
  title={Leum-VL Technical Report},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2603.20354}
}
```

- arXiv: 2603.20354