Csvqa Eval

---
name: csvqa-eval
description: This benchmark evaluates the scientific reasoning and domain-grounded visual question answering capabilities of Vision-Language Models (VLMs) in Chinese. It probes the ability to integrate multimodal STEM evidence across physics, chemistry, biology, and mathematics with domain knowledge to solve both multiple-choice and open-ended questions. Use when the user wants to benchmark on CSVQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.24120
  bibtex_key: ai2025csvqa
  confidence: high
---

# csvqa-eval

> CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs — Ai Jian et al. (arXiv:2505.24120, 2025)

## What this evaluates

This benchmark evaluates the scientific reasoning and domain-grounded visual question answering capabilities of Vision-Language Models (VLMs) in Chinese. It probes the ability to integrate multimodal STEM evidence across physics, chemistry, biology, and mathematics with domain knowledge to solve both multiple-choice and open-ended questions.

## Datasets

- **CSVQA** — total 1378; splits: test (1378)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Calculated as (number of correct predictions / total number of questions) × 100. Evaluated overall and broken down by subject (Biology, Chemistry, Math, Physics) and question type (Open-ended, Multiple-Choice).

## Input / output format

**Input**: A STEM-related image paired with a Chinese-language question. Questions are either multiple-choice (with one or more correct options, but the model is not told how many) or open-ended.

**Output**: Multiple-choice: fixed response format parsed deterministically, with a GPT-4o fallback if parsing fails. Open-ended: free-form text response, later scored by GPT-4o.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers, question_types):
    correct = 0
    for pred, gold, qtype in zip(predictions, gold_answers, question_types):
        if qtype == 'MC':
            parsed = deterministic_parser(pred)
            if parsed is None:
                parsed = gpt4o_fallback_match(pred, gold)
            if parsed == gold:
                correct += 1
        elif qtype == 'Open':
            if gpt4o_judge_consistency(pred, gold):
                correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Models are not informed whether multiple-choice questions have single or multiple correct answers, leading to format confusion and parser failures.
- Open-ended questions are not scored via exact string match; they require an external LLM (GPT-4o) for evaluation, which introduces potential judge bias and variability.
- Deterministic parsing of MC answers may fail on format violations, requiring a GPT-4o fallback mechanism to ensure consistent scoring across all models.

## Evidence (verbatim from paper)

> For multiple-choice questions, models are constrained to follow a fixed response format using rule-based protocols, with answers extracted via a deterministic parser. Notably, while questions may have single or multiple correct answers, models receive no explicit indication of the number of valid options. If the parser fails due to format violations, we employ a fallback mechanism using GPT-4o to match the answers. Open-ended questions are scored only by GPT-4o to ensure consistency and accuracy of scoring. Table 3 shows that the best-performing model on CSVQA is the closed-source o1, achieving an overall accuracy of 49.6%.

## Citation

```bibtex
@misc{ai2025csvqa,
  title={CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs},
  author={Ai Jian et al.},
  year={2025},
  note={arXiv:2505.24120}
}
```

- arXiv: 2505.24120