Towervision Multilingual Vl Eval

---
name: towervision-multilingual-vl-eval
description: This evaluation probes the multilingual vision-language capabilities of models across text recognition, cultural understanding, multimodal translation, and video reasoning. It specifically tests cross-lingual generalization and cultural grounding in both image and video domains across high- and low-resource languages. Use when the user wants to benchmark on ALM-Bench, OCRBench, cc-OCR, TextVQA, CoMMuTE, Multi30K, ViMUL-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.21849
  bibtex_key: viveiros2025towervision
  confidence: high
---

# towervision-multilingual-vl-eval

> TowerVision: Understanding and Improving Multilinguality in Vision-Language Models — Viveiros et al. (2025) (arXiv:2510.21849, 2025)

## What this evaluates

This evaluation probes the multilingual vision-language capabilities of models across text recognition, cultural understanding, multimodal translation, and video reasoning. It specifically tests cross-lingual generalization and cultural grounding in both image and video domains across high- and low-resource languages.

## Datasets

- **ALM-Bench** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **cc-OCR** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **CoMMuTE** — total ?; splits: test (-1)
- **Multi30K** — total ?; splits: test (-1)
- **ViMUL-Bench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correct predictions on closed-form or multiple-choice questions. Computed as (correct predictions / total instances) * 100.
- `xComet` — range: [0, 1]
  - Cross-lingual quality metric used for Multi30K translation evaluation. Scores translation quality on a normalized scale.
- `contrastive pairwise accuracy` — range: [0, 1]
  - Measures whether the correct translation scores higher than incorrect alternatives in CoMMuTE.
- `GPT-4o judge score` — range: [0, 1]
  - Automated LLM-as-judge scoring for open-ended ViMUL-Bench responses, averaged with multiple-choice accuracy.

## Input / output format

**Input**: Image or video frames paired with text prompts/questions in various languages (English and 20+ other languages).

**Output**: Text predictions (answers, translations, or captions). Open-ended responses are evaluated via LLM-as-judge.

## Scoring recipe

```python
def compute_metric(predictions, gold, task):
    if task in ['ALM-Bench', 'TextVQA', 'OCRBench', 'cc-OCR', 'ViMUL-Bench-MC']:
        return 1.0 if predictions == gold else 0.0
    elif task == 'Multi30K':
        return xcomet_score(predictions, gold)
    elif task == 'CoMMuTE':
        return contrastive_pairwise_accuracy(predictions, gold)
    elif task == 'ViMUL-Bench-Open':
        return gpt4o_judge_score(predictions, gold, prompt=shafique2025_prompt)
    return average(scores)
```

## Common pitfalls

- Assuming all benchmarks use closed-form evaluation; ViMUL-Bench open-ended responses require LLM-as-judge scoring.
- Overlooking language-specific splits; benchmarks like ALM-Bench and cc-OCR have distinct English vs. multilingual subsets that must be evaluated separately.
- Confusing xComet with standard BLEU/chrF for Multi30K; the paper explicitly uses xComet for cross-lingual quality.

## Evidence (verbatim from paper)

> We report xComet *(guerreiro2024xcomet)* for Multi30K and contrastive pairwise accuracy for CoMMuTE.

## Citation

```bibtex
@misc{viveiros2025towervision,
  title={TowerVision: Understanding and Improving Multilinguality in Vision-Language Models},
  author={Viveiros et al. (2025)},
  year={2025},
  note={arXiv:2510.21849}
}
```

- arXiv: 2510.21849