Miss Qa Eval

---
name: miss-qa-eval
description: Evaluates multimodal foundation models' ability to interpret schematic diagrams in scientific papers and answer information-seeking questions based on visual-textual context. It also probes models' robustness in identifying unanswerable questions when sufficient information is absent. Use when the user wants to benchmark on MISS-QA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.10787
  bibtex_key: zhao2025missqa
  confidence: high
---

# miss-qa-eval

> Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers — Zhao et al. (2025) (arXiv:2507.10787, 2025)

## What this evaluates

Evaluates multimodal foundation models' ability to interpret schematic diagrams in scientific papers and answer information-seeking questions based on visual-textual context. It also probes models' robustness in identifying unanswerable questions when sufficient information is absent.

## Datasets

- **MISS-QA** — total 1500; splits: testmini (500), test (1000)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Computed via an LLM-as-Judge framework (GPT-4.1) that compares the model's generated response to the ground-truth answer, assigning a score of 0, 0.5, or 1 per instance. The final metric is the average score across all instances.

## Input / output format

**Input**: A schematic diagram image, a natural language question, and the relevant textual context from the scientific paper.

**Output**: A natural language text response.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    scores = []
    for pred, gold in zip(predictions, gold_answers):
        # GPT-4.1 LLM-as-Judge compares pred and gold
        score = llm_judge(prompt=f'Compare response: {pred} with ground truth: {gold}')
        # Returns 0, 0.5, or 1
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Models frequently exhibit overconfidence on unanswerable questions, generating responses instead of correctly indicating that the information is missing.
- Models often over-rely on isolated visual interpretation of diagrams while neglecting the provided paper context, or fail to interpret diagram structures correctly.

## Evidence (verbatim from paper)

> We use accuracy as the primary metric to evaluate model performance on MISS-QA. Following recent benchmark studies (Lu et al., 2024; Li et al., 2024a), we adopt the LLM-as-Judge framework with GPT-4.1 as the base evaluator. Specifically, the model is prompted to assign an accuracy score of 0, 0.5, or 1 by comparing the generated response with the ground-truth answer.

## Citation

```bibtex
@misc{zhao2025missqa,
  title={Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2507.10787}
}
```

- arXiv: 2507.10787