M2k Vdg Eval

---
name: m2k-vdg-eval
description: Evaluates a model's ability to generate fluent, semantically accurate, and hallucination-free natural language responses grounded in video and audio inputs. It probes multimodal fusion, knowledge grounding, and dialogue generation capabilities across diverse question types and modalities. Use when the user wants to benchmark on AVSD10, NExT-OE, MUSIC-AVQA, or asks about evaluating this task. Reports CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.11875
  bibtex_key: liu2024m2kvdg
  confidence: high
---

# m2k-vdg-eval

> M2K-VDG: Model-Adaptive Multimodal Knowledge Anchor Enhanced Video-grounded Dialogue Generation — Hongcheng Liu et al. (2024) (arXiv:2402.11875, 2024)

## What this evaluates

Evaluates a model's ability to generate fluent, semantically accurate, and hallucination-free natural language responses grounded in video and audio inputs. It probes multimodal fusion, knowledge grounding, and dialogue generation capabilities across diverse question types and modalities.

## Datasets

- **AVSD10** — total ?; splits: test (-1)
- **NExT-OE** — total ?; splits: test (-1)
- **MUSIC-AVQA** — total ?; splits: test (-1)

## Metrics

- `BLEU1-4` — range: [0, 1]
  - N-gram precision scores (1-gram to 4-gram) comparing generated text to reference responses. BLEU4 is the standard n-gram overlap metric for text generation.
- `METEOR` — range: [0, 1]
  - Metric for Evaluation of Translation with Explicit ORdering, weighing precision, recall, and fragmentation penalty based on synonym and stem matching.
- `ROUGH-L` — range: [0, 1]
  - Longest Common Subsequence (LCS) based recall/precision score. Note: The paper misspells this as ROUGH-L instead of ROUGE-L.
- `CIDEr` **(primary)** — range: [0, 1]
  - Consensus-based Image Description Evaluation. Computes TF-IDF weighted n-gram similarity between generated and reference captions, emphasizing consensus with human references.
- `WUPS Score` — range: [0, 1]
  - Weighted Unambiguous Precision and Recall Score. Measures semantic similarity based on word overlap aligned via a knowledge hierarchy (e.g., WordNet).
- `Accuracy` — range: [0, 1]
  - Exact match or fuzzy match rate for question-answering tasks, measuring the percentage of correctly generated answers against ground truth.

## Input / output format

**Input**: Multimodal context consisting of video frames/clips and corresponding audio tracks, paired with a conversational history or specific question.

**Output**: A single natural language sentence or short paragraph representing the model's generated dialogue response.

## Scoring recipe

```python
def evaluate(predictions, references):
    bleu4 = nltk.bleu([refs], preds, weights=(0.25, 0.25, 0.25, 0.25))
    meteor = compute_meteor(references, predictions)
    rouge_l = compute_rouge_l(references, predictions)  # Paper calls it ROUGH-L
    cider = compute_cider(references, predictions)
    wups = compute_wups(references, predictions)
    acc = sum(1 for p, r in zip(predictions, references) if match(p, r)) / len(predictions)
    return {'BLEU4': bleu4, 'METEOR': meteor, 'ROUGE-L': rouge_l, 'CIDEr': cider, 'WUPS': wups, 'Accuracy': acc}
```

## Common pitfalls

- The paper misspells ROUGE-L as 'ROUGH-L' throughout the text and tables.
- Metrics are reported across three distinct datasets (AVSD10, NExT-OE, MUSIC-AVQA) with different modalities and question types; results cannot be averaged or directly compared across datasets.
- Accuracy for MUSIC-AVQA does not specify whether exact match or fuzzy matching is used, which significantly impacts reported scores.
- WUPS Score requires a specific knowledge hierarchy (e.g., WordNet) for alignment; results vary heavily depending on the implementation version.

## Evidence (verbatim from paper)

> Furthermore, we adopt different evaluation metrics from various aspects, including fluent expression (BLEU1-4, METEOR, ROUGH-L, CIDEr), semantic representation (WUPS Score $^{1}$ ), and accurate generation (Accuracy).

## Citation

```bibtex
@misc{liu2024m2kvdg,
  title={M2K-VDG: Model-Adaptive Multimodal Knowledge Anchor Enhanced Video-grounded Dialogue Generation},
  author={Hongcheng Liu et al. (2024)},
  year={2024},
  note={arXiv:2402.11875}
}
```

- arXiv: 2402.11875