Univa Bench Eval

---
name: univa-bench-eval
description: Evaluates a video agent's capabilities across generation, understanding, editing, and segmentation tasks, while probing its agentic planning and memory mechanisms. It measures how well a unified agent architecture handles long-horizon, multi-step video workflows compared to monolithic baselines. Use when the user wants to benchmark on UniVA-Bench, or asks about evaluating this task. Reports MLLM Judge.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.08521
  bibtex_key: liang2025univa
  confidence: high
---

# univa-bench-eval

> UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist — Zhengyang Liang et al. (arXiv:2511.08521, 2025)

## What this evaluates

Evaluates a video agent's capabilities across generation, understanding, editing, and segmentation tasks, while probing its agentic planning and memory mechanisms. It measures how well a unified agent architecture handles long-horizon, multi-step video workflows compared to monolithic baselines.

## Datasets

- **UniVA-Bench** — total ?; splits: test (-1)

## Metrics

- `CLIP Score` — range: [0, 1]
  - Measures prompt following by computing the cosine similarity between the text prompt embedding and the generated video embedding using a CLIP model.
- `DINO Score` — range: [0, 1]
  - Measures subject consistency by computing the cosine similarity between reference image/video embeddings and generated video embeddings using a DINO model.
- `MLLM Judge` **(primary)** — range: other
  - Preference ratings assigned by a Multimodal Large Language Model acting as a judge, evaluating generated videos against a fixed set of criteria aligned with human preferences.
- `Normalized QA Accuracy` — range: [0, 1]
  - The proportion of correctly answered questions in long video understanding tasks, normalized according to the UniVA-Bench protocol.
- `J&F-mean` — range: [0, 1]
  - The harmonic mean of J-mean (Jaccard index) and F-mean (frame similarity) for video segmentation evaluation.
- `Success Rate` — range: [0, 1]
  - The percentage of test cases where the agent produced a structurally valid plan, defined as cases where wPED > 0.
- `wPED` — range: [0, 1]
  - Weighted Plan Execution Distance; a metric quantifying the quality of generated plans relative to gold plans, where lower values indicate better plan quality.

## Input / output format

**Input**: Text prompts, reference images, source videos, and long multi-turn instructions or queries.

**Output**: Generated videos, QA answers, segmentation masks, and structured planning trajectories/step sequences.

## Scoring recipe

```python
def score_generation(prompt, ref, video):
    clip = compute_clip_similarity(prompt, video)
    dino = compute_dino_similarity(ref, video)
    mllm = mllm_judge.evaluate(video, prompt, criteria)
    return clip, dino, mllm

def score_understanding(pred_answers, gold_answers):
    return normalized_accuracy(pred_answers, gold_answers)

def score_segmentation(pred_masks, gt_masks):
    j = j_mean(pred_masks, gt_masks)
    f = f_mean(pred_masks, gt_masks)
    return (j + f) / 2

def score_planning(plans, gold_plans):
    success_rate = mean([1 if wPED(plan, gold) > 0 else 0 for plan, gold in zip(plans, gold_plans)])
    wped_scores = [wPED(plan, gold) for plan, gold in zip(plans, gold_plans)]
    return success_rate, wped_scores
```

## Common pitfalls

- Automated metrics like CLIP and DINO may penalize videos that successfully follow complex narrative instructions but differ in strict frame-level similarity.
- MLLM-as-a-Judge scores require careful calibration against human preferences, though the paper claims strong alignment.
- wPED, DepCov, and ReplanQ metrics are referenced but their exact mathematical formulations are deferred to the Appendix or benchmark specification.
- Baseline model setups and hyperparameters are not detailed in the main text, requiring Appendix [9] for full reproducibility.

## Evidence (verbatim from paper)

> Evaluating the results using CLIP Score (prompt following), DINO Score (subject consistency), and preference ratings from an MLLM-as-a-Judge, following the UniVA-Bench specification. Performance is measured by the normalized QA accuracy score as defined in the UniVA-Bench protocol.

## Citation

```bibtex
@misc{liang2025univa,
  title={UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist},
  author={Zhengyang Liang et al.},
  year={2025},
  note={arXiv:2511.08521}
}
```

- arXiv: 2511.08521