Video Qa Eval

---
name: video-qa-eval
description: Evaluates video large multimodal models on question-answering tasks across multiple benchmark datasets. Probes the model's ability to understand video content, generate factually accurate long-form responses, and align with language model-derived preferences using direct preference optimization. Use when the user wants to benchmark on MSVD-QA, MSRVTT-QA, TGIF-QA, ActivityNet-QA, VIDAL-QA, WebVid-QA, SSV2-QA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.01258
  bibtex_key: zhang2024directpreferenceoptimization
  confidence: high
---

# video-qa-eval

> Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward — Ruohong Zhang et al. (2024) (arXiv:2404.01258, 2024)

## What this evaluates

Evaluates video large multimodal models on question-answering tasks across multiple benchmark datasets. Probes the model's ability to understand video content, generate factually accurate long-form responses, and align with language model-derived preferences using direct preference optimization.

## Datasets

- **MSVD-QA** — total ?; splits: test (-1)
- **MSRVTT-QA** — total ?; splits: test (-1)
- **TGIF-QA** — total ?; splits: test (-1)
- **ActivityNet-QA** — total ?; splits: test (-1)
- **VIDAL-QA** — total ?; splits: test (-1)
- **WebVid-QA** — total ?; splits: test (-1)
- **SSV2-QA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Correctness is determined by ChatGPT evaluation against ground truth (existing benchmarks) or by a ChatGPT-assigned quality score ≥ 3 (proposed benchmarks).
- `score` — range: other
  - Continuous quality score (approximately 1-5 scale) assigned by ChatGPT based on response relevance, accuracy, clarity, and completeness when evaluated against GPT-4V generated captions as supporting evidence.

## Input / output format

**Input**: Video clip (or sequence of frames) paired with a natural language question.

**Output**: Free-form text response answering the question.

## Scoring recipe

```python
def evaluate(model, video, question, ground_truth=None, use_proposed=False):
    if use_proposed:
        evidence = get_gpt4v_caption(video)
        prompt = f'Q: {question}\nEvidence: {evidence}\nPred: {model(video, question)}\nScore 1-5:'
        score = call_chatgpt(prompt, model='gpt-3.5-turbo-0301')
        acc = 1 if score >= 3 else 0
        return acc, score
    else:
        prompt = f'Q: {question}\nGT: {ground_truth}\nPred: {model(video, question)}\nCorrect?'
        is_correct = call_chatgpt(prompt, model='gpt-3.5-turbo-0613')
        return is_correct
```

## Common pitfalls

- Different ChatGPT versions significantly impact absolute metric values, though overall model rankings remain relatively stable.
- Existing benchmarks use auto-generated questions that may be grammatically incorrect or nonsensical, and single-word ground truths that fail to evaluate long-form generation quality.
- Unfreezing the MLP projector during DPO training causes a notable performance drop compared to keeping it frozen.

## Evidence (verbatim from paper)

> We evaluate model performance on three benchmark datasets: MSVD-QA Chen & Dolan (2011), MSRVTT-QA Xu et al. (2016), and TGIF-QA Jang et al. (2017), using ChatGPT with version gpt-3.5-turbo-0611 to assess model predictions. ... Applying the our reward system in section 4, we report the score from ChatGPT, and a score value ≥ 3 will be considered correct for accuracy calculation.

## Citation

```bibtex
@misc{zhang2024directpreferenceoptimization,
  title={Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward},
  author={Ruohong Zhang et al. (2024)},
  year={2024},
  note={arXiv:2404.01258}
}
```

- arXiv: 2404.01258