Seam Eval

---
name: seam-eval
description: Evaluates vision-language models on their ability to reason across semantically equivalent inputs presented in different modalities (vision vs. language) across domain-specific notation systems. It probes cross-modal consistency, modality-agnostic reasoning, and identifies domain-specific perception and tokenization failure modes. Use when the user wants to benchmark on SEAM, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.18179
  bibtex_key: tang2025seam
  confidence: high
---

# seam-eval

> SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models — Tang et al. (2025) (arXiv:2508.18179, 2025)

## What this evaluates

Evaluates vision-language models on their ability to reason across semantically equivalent inputs presented in different modalities (vision vs. language) across domain-specific notation systems. It probes cross-modal consistency, modality-agnostic reasoning, and identifies domain-specific perception and tokenization failure modes.

## Datasets

- **SEAM** — total ?; splits: test (-1); repo https://github.com/CSSLab/SEAM

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered questions out of the total number of samples.
- `agreement_rate` — range: [0, 1]
  - Binary agreement between vision and language answers for each sample (1 if identical, 0 otherwise), averaged across all samples.

## Input / output format

**Input**: Zero-shot chain-of-thought prompt containing a question with domain-specific notation (chess FEN, chemistry SMILES, music ABC, or graph adjacency matrix) presented either as plain text or as an image.

**Output**: Free-text final answer extracted from the model's generation, typically a single letter/option or short string, post-processed via an external LLM (Qwen2.5-7B-Instruct) using the OpenCompass protocol when rule-based extraction fails.

## Scoring recipe

```python
def compute_metrics(preds_vision, preds_language, gold):
    accuracy = sum(1 for p in preds_vision if p == gold) / len(gold)
    agreements = [1 if v == l else 0 for v, l in zip(preds_vision, preds_language)]
    agreement_rate = sum(agreements) / len(agreements)
    return accuracy, agreement_rate
```

## Common pitfalls

- High accuracy artifactually inflates cross-modal agreement because both modalities converge on the correct answer; true alignment requires comparing agreement against a random baseline (p^2 + (1-p)^2/3).
- Assuming vision inputs compensate for language tokenization errors; the paper shows vision can actually degrade performance in domains where text parsing is difficult.
- Ignoring domain-specific tokenization limitations in text modalities (e.g., SMILES, FEN) which independently drive performance gaps unrelated to visual perception.

## Evidence (verbatim from paper)

> For each sample, the agreement between modalities is binary (either 0 for disagree or 1 for agree), and these binary values are then averaged across all samples to obtain the overall agreement rate. ... GPT-5 demonstrating superior accuracy (0.765) compared to the highest-performing open-source model, Qwen2.5-VL-72B-Instruct (0.514).

## Citation

```bibtex
@misc{tang2025seam,
  title={SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models},
  author={Tang et al. (2025)},
  year={2025},
  note={arXiv:2508.18179}
}
```

- arXiv: 2508.18179