Mudaif Vl Eval

---
name: mudaif-vl-eval
description: Evaluates a decoder-only vision-language model's ability to perform visual question answering, image captioning, and multimodal reasoning. It measures cross-modal alignment, computational efficiency, and robustness to input variations like resolution and noise. Use when the user wants to benchmark on VQA-v2, GQA, VizWiz, SEED, MM-Vet, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.10758
  bibtex_key: tanaka2024mudaif
  confidence: high
---

# mudaif-vl-eval

> Optimizing Vision-Language Interactions Through Decoder-Only Models — Tanaka et al. (2024) (arXiv:2412.10758, 2024)

## What this evaluates

Evaluates a decoder-only vision-language model's ability to perform visual question answering, image captioning, and multimodal reasoning. It measures cross-modal alignment, computational efficiency, and robustness to input variations like resolution and noise.

## Datasets

- **VQA-v2** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **VizWiz** — total ?; splits: test (-1)
- **SEED** — total ?; splits: test (-1)
- **MM-Vet** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of questions.
- `BLEU` — range: [0, 1]
  - Standard n-gram overlap metric for evaluating image captioning quality.
- `SEED Score` — range: other
  - Composite score for multimodal reasoning benchmarks.
- `MM-Vet Score` — range: other
  - Composite score evaluating multimodal capabilities and instruction following.

## Input / output format

**Input**: Paired image and text input (question or instruction prompt).

**Output**: Text string containing the predicted answer or caption.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- SEED and MM-Vet use proprietary or complex composite scoring rubrics that are not publicly detailed in this paper; users must use official evaluation scripts.
- BLEU is reported for captioning but the specific dataset and n-gram order are not specified, requiring standard implementation assumptions.
- Human evaluation uses a 1-5 Likert scale averaged across tasks, which may introduce rater bias and lacks inter-annotator agreement metrics.

## Evidence (verbatim from paper)

> Metrics include accuracy for VQA, BLEU for captioning, and composite scores for multimodal benchmarks. Human evaluation assesses relevance, coherence, and informativeness.

## Citation

```bibtex
@misc{tanaka2024mudaif,
  title={Optimizing Vision-Language Interactions Through Decoder-Only Models},
  author={Tanaka et al. (2024)},
  year={2024},
  note={arXiv:2412.10758}
}
```

- arXiv: 2412.10758