Vqa Captioning Eval

---
name: vqa-captioning-eval
description: Evaluates multimodal language models on visual question answering and image captioning tasks, probing their zero-shot and few-shot in-context learning capabilities with interleaved image-text inputs. Use when the user wants to benchmark on OKVQA, TextVQA, COCO, Flickr30k, VQAv2, VizWiz, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.08418
  bibtex_key: li2024omnicorpus
  confidence: high
---

# vqa-captioning-eval

> OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text — Qingyun Li et al. (2024) (arXiv:2406.08418, 2024)

## What this evaluates

Evaluates multimodal language models on visual question answering and image captioning tasks, probing their zero-shot and few-shot in-context learning capabilities with interleaved image-text inputs.

## Datasets

- **OKVQA** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **COCO** — total ?; splits: val/test (-1)
- **Flickr30k** — total ?; splits: test (-1)
- **VQAv2** — total ?; splits: test (-1)
- **VizWiz** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers for VQA tasks. Calculated as the number of exact matches between predicted and ground-truth answers divided by the total number of samples.
- `CIDEr` — range: [0, 1]
  - Consensus-based Image Description Evaluation metric. Computes TF-IDF weighted n-gram similarity between generated captions and reference captions, averaging across n-grams of length 1 to 8.

## Input / output format

**Input**: Interleaved sequence of visual and textual tokens. For few-shot evaluation, the input includes in-context examples sampled using RICES.

**Output**: Text string containing the predicted answer (for VQA) or generated caption (for image captioning).

## Scoring recipe

```python
def score_vqa(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == normalize_answer(g))
    return correct / len(gold)

def score_captioning(predictions, gold):
    # Use standard CIDEr implementation
    return compute_cider(predictions, gold)
```

## Common pitfalls

- Few-shot in-context examples are sampled using RICES, not randomly, which significantly impacts reproducibility if not specified.
- Zero-shot evaluation for VQAv2 and VizWiz actually includes two text-only examples following the Flamingo protocol, so it is not pure zero-shot.
- The TextVQA prompt explicitly excludes OCR tokens, differing from standard OCR-augmented VQA evaluation settings.

## Evidence (verbatim from paper)

> The accuracy score is used for VQA, while CIDEr [108] is used for image captioning. Following OpenFlamingo [3], we extend the benchmarks to few-shot settings to assess in-context learning. Specifically, in-context examples are sampled using RICES [118].

## Citation

```bibtex
@misc{li2024omnicorpus,
  title={OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text},
  author={Qingyun Li et al. (2024)},
  year={2024},
  note={arXiv:2406.08418}
}
```

- arXiv: 2406.08418