Sea Vision Eval

---
name: sea-vision-eval
description: Evaluates multimodal language models on document parsing and text-centric visual question answering across 11 Southeast Asian languages. Probes the models' ability to extract structured information from complex documents and answer questions based on visual-textual alignment in low-resource scripts. Use when the user wants to benchmark on SEA-Vision, or asks about evaluating this task. Reports answer accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15409
  bibtex_key: [REDACTED]
  confidence: high
---

# sea-vision-eval

> SEA-Vision: A Multilingual Benchmark for Comprehensive Document and Scene Text Understanding in Southeast Asia — Yue et al. (2026) (arXiv:2603.15409, 2026)

## What this evaluates

Evaluates multimodal language models on document parsing and text-centric visual question answering across 11 Southeast Asian languages. Probes the models' ability to extract structured information from complex documents and answer questions based on visual-textual alignment in low-resource scripts.

## Datasets

- **SEA-Vision** — total 22730; splits: test (-1)

## Metrics

- `answer accuracy` **(primary)** — range: percent
  - Percentage of questions where the model's raw textual output exactly matches the ground truth answer after standard normalization. Higher values indicate better performance.
- `NED` — range: [0, 1]
  - Normalized Edit Distance measuring the structural and textual alignment between predicted and ground-truth document parsing outputs. Lower values indicate better performance.

## Input / output format

**Input**: An image of a document or scene text page, optionally accompanied by a natural language question for the TEC-VQA task.

**Output**: For document parsing: a structured representation of OCR, tables, formulas, and reading order. For TEC-VQA: a raw textual answer string.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
    return (correct / len(golds)) * 100

def compute_ned(pred_structure, gold_structure):
    dist = edit_distance(pred_structure, gold_structure)
    max_len = max(len(pred_structure), len(gold_structure))
    return dist / max_len if max_len > 0 else 0.0
```

## Common pitfalls

- Models are evaluated strictly in a zero-shot setting using a unified prompt template; comparing against fine-tuned or task-specific baselines violates the evaluation protocol.
- Raw textual outputs are used directly for scoring without post-processing or parsing, which may unfairly penalize models that output structured formats (e.g., JSON, Markdown) differently than the ground truth.
- NED is a structural metric where lower is better, contrary to standard accuracy metrics where higher is better, which can lead to misinterpretation of results.

## Evidence (verbatim from paper)

> For document parsing, we report NED↓ where lower is better. For TEC-VQA (Text-centric Visual Question Answering), we report answer accuracy↑ where higher is better. To ensure a fair comparison, all models are assessed in a unified, zero-shot setting using the same prompt template and the answer accuracy is computed directly from their raw textual outputs without any task-specific fine-tuning.

## Citation

```bibtex
@misc{yue2026seavision,
  title={SEA-Vision: A Multilingual Benchmark for Comprehensive Document and Scene Text Understanding in Southeast Asia},
  author={Yue et al. (2026)},
  year={2026},
  note={arXiv:2603.15409}
}
```

- arXiv: 2603.15409