M3it Eval

---
name: m3it-eval
description: Evaluates a vision-language model's ability to follow multi-modal instructions, answer knowledge-based visual questions, and generalize to unseen languages and video tasks. It probes cross-modal alignment, cross-lingual transfer, and the model's conversational response quality. Use when the user wants to benchmark on M^3IT, OK-VQA, A-OKVQA, ViQuAE, Flickr-8k-CN, FM-IQA, Chinese-FoodNet, MSRVTT, iVQA, ActivityNet-QA, MSRVTT-QA, MSVD-QA, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.04387
  bibtex_key: li2023m3it
  confidence: high
---

# m3it-eval

> M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning — Lei Li et al. (2023) (arXiv:2306.04387, 2023)

## What this evaluates

Evaluates a vision-language model's ability to follow multi-modal instructions, answer knowledge-based visual questions, and generalize to unseen languages and video tasks. It probes cross-modal alignment, cross-lingual transfer, and the model's conversational response quality.

## Datasets

- **M^3IT** — total 2400000; splits: train (-1), test (-1)
- **OK-VQA** — total ?; splits: test (-1)
- **A-OKVQA** — total ?; splits: test (-1)
- **ViQuAE** — total ?; splits: test (-1)
- **Flickr-8k-CN** — total ?; splits: test (-1)
- **FM-IQA** — total ?; splits: test (-1)
- **Chinese-FoodNet** — total ?; splits: test (-1)
- **MSRVTT** — total ?; splits: test (-1)
- **iVQA** — total ?; splits: test (-1)
- **ActivityNet-QA** — total ?; splits: test (-1)
- **MSRVTT-QA** — total ?; splits: test (-1)
- **MSVD-QA** — total ?; splits: test (-1)

## Metrics

- `ROUGE-L` **(primary)** — range: [0, 1]
  - Computes the longest common subsequence (LCS) based F1 score between the predicted text and the ground-truth answer, focusing on recall of the reference.
- `GPT-4 rating` — range: [1, 10]
  - GPT-4 rates model responses on a scale of 1-10 based on accuracy, relevance, and naturalness against a reference answer. Winning rate is calculated as the percentage of samples where the model outperforms a baseline.

## Input / output format

**Input**: Image (or uniformly sampled 8 frames for video) paired with a text instruction or question.

**Output**: Text response or answer generated by the model.

## Scoring recipe

```python
# ROUGE-L calculation
rouge_l = compute_rouge_l(predictions=pred_text, references=gold_text)

# GPT-4 proxy evaluation
prompt = f"Question: {q}\nReference: {ref}\nModel A: {pred}\nModel B: {baseline}\nRate 1-10 on accuracy, relevance, naturalness."
rating = gpt4_api.query(prompt).score
winning_rate = sum(1 for r in ratings if r['model'] > r['baseline']) / len(ratings) * 100
```

## Common pitfalls

- ROUGE-L focuses on n-gram overlap and may penalize semantically correct but lexically different responses, failing to capture full conversational quality.
- GPT-4 evaluation is based on only 300 randomly sampled examples, which may not generalize to the full test set distribution.
- Video tasks require uniform sampling of 8 frames, which is a specific preprocessing step that affects temporal context and must be replicated exactly.

## Evidence (verbatim from paper)

> We adopt ROUGE-L [26] as an automatic metric to assess the consistency between predictions and ground-truth answers, focusing on evaluating the model's conversational abilities. As the automatic metric may not fully capture the nuances of conversational quality, we further introduce GPT-4 as a proxy of human evaluators (§ 4.2).

## Citation

```bibtex
@misc{li2023m3it,
  title={M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning},
  author={Lei Li et al. (2023)},
  year={2023},
  note={arXiv:2306.04387}
}
```

- arXiv: 2306.04387