Video Captioning Eval

---
name: video-captioning-eval
description: Evaluates fine-grained audiovisual captioning quality, attribute-level instruction following, and downstream reasoning capabilities like QA and temporal grounding. Use when the user wants to benchmark on video-SALMONN-2, UGC-VideoCap, VDC, VidCapBench-AE, Daily-Omni, World-Sense, Charades-STA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.13013
  bibtex_key: li2026towards
  confidence: high
---

# video-captioning-eval

> Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions — Li et al. (2026) (arXiv:2602.13013, 2026)

## What this evaluates

Evaluates fine-grained audiovisual captioning quality, attribute-level instruction following, and downstream reasoning capabilities like QA and temporal grounding.

## Datasets

- **video-SALMONN-2** — total ?; splits: test (-1)
- **UGC-VideoCap** — total ?; splits: test (-1)
- **VDC** — total ?; splits: test (-1)
- **VidCapBench-AE** — total ?; splits: test (-1)
- **Daily-Omni** — total ?; splits: test (-1)
- **World-Sense** — total ?; splits: test (-1)
- **Charades-STA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered QA questions or correctly followed attribute instructions.
- `mIoU` — range: [0, 1]
  - Mean Intersection over Union between predicted and ground-truth temporal segments.
- `R1@τ` — range: [0, 1]
  - Recall at IoU threshold τ, measuring the fraction of predictions where IoU ≥ τ.
- `Miss/Hall/Total` — range: [0, 1]
  - Rates of missing details, hallucinations, and total errors in captions, respectively.
- `Acc/Pre/Cov/Con` — range: [0, 100]
  - Accuracy, precision, coverage, and conciseness scores for text-to-video generation captions, scaled by 100.

## Input / output format

**Input**: Video frames/audio, optionally accompanied by attribute instructions or QA prompts.

**Output**: Text captions (optionally timestamped and attribute-structured), QA answers, or temporal start/end timestamps.

## Scoring recipe

```python
def score(predictions, gold):
    acc = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    ious = [iou(p.start, p.end, g.start, g.end) for p, g in zip(predictions, gold)]
    miou = sum(ious) / len(ious)
    r1_tau = sum(1 for i in ious if i >= tau) / len(ious)
    miss = sum(1 for p in predictions if p.flags.missing) / len(predictions)
    hall = sum(1 for p in predictions if p.flags.hallucination) / len(predictions)
    return acc, miou, r1_tau, miss, hall
```

## Common pitfalls

- LLM-based judges (GPT-4o, Gemini-2.5-Pro) are used for scoring, which may introduce model-specific biases.
- Hallucination metrics penalize unannotated but relevant details (e.g., on-screen text) as errors.
- Temporal grounding relies on caption-only prompts to a fixed QA model, which may not reflect end-to-end model performance.

## Evidence (verbatim from paper)

> We evaluate ASID-Captioner on seven benchmarks that offer complementary perspectives on caption quality. ... Start and end timestamps of the queried moment are predicted using only the caption, and we report mIoU and recall at IoU thresholds (R1@$	au$).

## Citation

```bibtex
@misc{li2026towards,
  title={Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2602.13013}
}
```

- arXiv: 2602.13013