Mvl Sib Eval

---
name: mvl-sib-eval
description: Evaluates cross-modal and text-only topical matching capabilities of vision-language models across 205 languages. It probes whether models can correctly associate images with semantically related texts (or vice versa) in a multilingual multiple-choice setting. Use when the user wants to benchmark on MVL-SIB, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.12852
  bibtex_key: schmidt2025mvl_sib
  confidence: high
---

# mvl-sib-eval

> MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching — Schmidt et al. (2025) (arXiv:2502.12852, 2025)

## What this evaluates

Evaluates cross-modal and text-only topical matching capabilities of vision-language models across 205 languages. It probes whether models can correctly associate images with semantically related texts (or vice versa) in a multilingual multiple-choice setting.

## Datasets

- **MVL-SIB** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Computed as the share of prompts for which the model's generated response begins with the correct ground-truth label letter (e.g., 'A', 'B'). Punctuation immediately following the letter (e.g., 'A.') is ignored.

## Input / output format

**Input**: An image (downsampled to 640×480 pixels) and a multilingual text prompt/question, presented as a multiple-choice task with labeled options (e.g., A, B, C).

**Output**: A single uppercase letter corresponding to the chosen option (e.g., 'A', 'B', 'C'), optionally followed by punctuation.

## Scoring recipe

```python
def compute_metric(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        first_char = next((c for c in pred if c.isalpha()), None)
        if first_char == gold:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Only the first alphabetic character of the model's output is scored; trailing explanations or punctuation are ignored.
- Images are strictly downsampled to 640×480 pixels, which may remove fine-grained visual details required for some topical matching tasks.
- Greedy decoding (temperature=0.0) is enforced, which may disadvantage models that perform better with stochastic sampling.

## Evidence (verbatim from paper)

> We compute the share of prompts for which responses begin with the right letter. If the label is "A", a response such as "A." is also correct.

## Citation

```bibtex
@misc{schmidt2025mvl_sib,
  title={MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching},
  author={Schmidt et al. (2025)},
  year={2025},
  note={arXiv:2502.12852}
}
```

- arXiv: 2502.12852