Hiscibench Eval

---
name: hiscibench-eval
description: Evaluates large language models across five hierarchical cognitive stages of scientific inquiry, ranging from foundational factual recall and literature parsing to advanced synthesis, literature review generation, and data-driven scientific discovery. It probes multimodal comprehension, cross-lingual reasoning, and computational problem-solving across six scientific disciplines. Use when the user wants to benchmark on HiSciBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.22899
  bibtex_key: zhang2025hiscibench
  confidence: medium
---

# hiscibench-eval

> HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery — Zhang et al. (2025) (arXiv:2512.22899, 2025)

## What this evaluates

Evaluates large language models across five hierarchical cognitive stages of scientific inquiry, ranging from foundational factual recall and literature parsing to advanced synthesis, literature review generation, and data-driven scientific discovery. It probes multimodal comprehension, cross-lingual reasoning, and computational problem-solving across six scientific disciplines.

## Datasets

- **HiSciBench** — total 8735; splits: test (8735)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Proportion of correctly answered instances out of the total. For multiple-choice and QA tasks (L1–L3), it is exact-match accuracy. For generation tasks (L4–L5), correctness is determined by matching against expert-annotated reference solutions or automated execution/judgment.

## Input / output format

**Input**: Varies by level: L1 provides multiple-choice questions; L2 provides multimodal scientific documents (images of pages, LaTeX/Markdown); L3 provides multimodal literature with questions; L4 provides research topic keywords and core papers; L5 provides problem descriptions, data files, and domain knowledge.

**Output**: L1–L3: selected option or generated answer; L4: comprehensive literature review text; L5: Python code, execution results, and interpretation.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if pred == gold_item:  # exact match for QA
            correct += 1
        elif is_correct_generation(pred, gold_item):  # L4/L5 expert/LLM judgment
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models must handle complex multimodal layouts (equations, figures) in L2, not just plain text.
- L5 requires executable Python code and data analysis, not just conceptual answers.
- Cross-lingual tasks (L2.2, L3.2) require preserving technical semantics and symbolic accuracy across languages.

## Evidence (verbatim from paper)

> state-of-the-art models achieve 69% accuracy on foundational tasks but drop to 25% on discovery-level reasoning, highlighting a critical gap in synthesis and creativity.

## Citation

```bibtex
@misc{zhang2025hiscibench,
  title={HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2512.22899}
}
```

- arXiv: 2512.22899