Omniscience Eval

---
name: omniscience-eval
description: Evaluates the semantic alignment and factual fidelity of automatically generated scientific image captions. It probes whether dense, context-aware captions can replace visual inputs for downstream reasoning tasks and how well they capture complex scientific figures compared to raw human-written captions. Use when the user wants to benchmark on OmniScience, AI2D, MMMU, MM-MT-Bench, MSEarth, or asks about evaluating this task. Reports cross-modal relevance score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.13758
  bibtex_key: [REDACTED]
  confidence: high
---

# omniscience-eval

> OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understanding — Tao et al. (2026) (arXiv:2602.13758, 2026)

## What this evaluates

Evaluates the semantic alignment and factual fidelity of automatically generated scientific image captions. It probes whether dense, context-aware captions can replace visual inputs for downstream reasoning tasks and how well they capture complex scientific figures compared to raw human-written captions.

## Datasets

- **OmniScience** — total ?; splits: test (-1), val (5000)
- **AI2D** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MM-MT-Bench** — total ?; splits: test (-1)
- **MSEarth** — total ?; splits: test (-1)

## Metrics

- `cross-modal relevance score` **(primary)** — range: continuous (higher is better)
  - Computed by Qwen3-VL-Reranker-8B, a single-tower cross-encoder that jointly encodes image-caption pairs and models cross-modal interactions via cross attention to yield instance-level semantic grounding scores.
- `LLM-as-a-Judge score` — range: [1, 5]
  - Mean of four 1–5 scale dimensions: Language Fluency, Information Consistency, Key Information Accuracy, and Detail Level. Scores from Qwen3-VL-235B-A22B-Thinking and Seed-1.5VL are averaged.
- `Caption QA accuracy` — range: [0, 1]
  - Accuracy of GPT-4o-mini answering VQA questions when the visual placeholder is replaced by a generated caption. Evaluated on AI2D, MMMU, MM-MT-Bench, and MSEarth.

## Input / output format

**Input**: Image + caption (for reranker); generated caption + reference caption + image (for LLM-Judge); question + generated caption (for Caption QA proxy task).

**Output**: Scalar relevance score (reranker); 1–5 score per dimension (LLM-Judge); correct/incorrect answer label (Caption QA).

## Scoring recipe

```python
def compute_cross_modal_relevance_score(images, captions):
    scores = []
    for img, cap in zip(images, captions):
        score = reranker_model.encode_cross_modal(img, cap)
        scores.append(score)
    return mean(scores)

def compute_llm_judge_score(captions, references, images):
    all_scores = []
    for cap, ref, img in zip(captions, references, images):
        scores = [judge.evaluate(cap, ref, img) for judge in [judge1, judge2]]
        all_scores.append(mean(scores))
    return mean(all_scores)

def compute_caption_qa_accuracy(questions, captions, gold_answers):
    correct = 0
    for q, cap, gold in zip(questions, captions, gold_answers):
        pred = reasoning_engine.answer(q, caption=cap)
        if pred == gold: correct += 1
    return correct / len(questions)
```

## Common pitfalls

- Using dual-tower embedding models (e.g., CLIP) instead of the specified cross-encoder reranker, which fails to capture fine-grained scientific figure details and yields less precise scores.
- Evaluating raw captions without article context, causing referential ambiguity (e.g., 'as shown in the left panel') and artificially low alignment scores compared to self-contained recaptioned text.
- Treating LLM-as-a-Judge scores as absolute ground truth without acknowledging they are relative rankings validated only on a 300-sample human-annotated subset (Kappa 0.831).

## Evidence (verbatim from paper)

> We employ Qwen3-VL-Reranker-8B to compute fine-grained relevance scores between images and captions. Compared to embedding based dual-tower similarity models, the reranker adopts a single-tower cross-encoder that jointly encodes image–caption pairs and explicitly models cross-modal interactions using cross attention, yielding more precise instance-level semantic grounding.

## Citation

```bibtex
@misc{tao2026omniscience,
  title={OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understanding},
  author={Tao et al. (2026)},
  year={2026},
  note={arXiv:2602.13758}
}
```

- arXiv: 2602.13758