Sc Heureka Bench Eval

---
name: sc-heureka-bench-eval
description: Evaluates AI co-scientist agents' ability to autonomously plan, execute, and interpret single-cell biology workflows to answer open-ended research questions (OEQs) and multiple-choice questions (MCQs). It probes hypothesis generation, code execution, data analysis, and scientific reasoning in a domain-specific setting. Use when the user wants to benchmark on sc-HeurekaBench-Lite, or asks about evaluating this task. Reports Correctness [1-5].
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.01678
  bibtex_key: panigrahi2026heurekabench
  confidence: high
---

# sc-heureka-bench-eval

> HeurekaBench: A Benchmarking Framework for AI Co-scientist — Panigrahi et al. (2026) (arXiv:2601.01678, 2026)

## What this evaluates

Evaluates AI co-scientist agents' ability to autonomously plan, execute, and interpret single-cell biology workflows to answer open-ended research questions (OEQs) and multiple-choice questions (MCQs). It probes hypothesis generation, code execution, data analysis, and scientific reasoning in a domain-specific setting.

## Datasets

- **sc-HeurekaBench-Lite** — total 40; splits: test (40); repo https://github.com/mlbio-epfl/heurekabench

## Metrics

- `Correctness [1-5]` **(primary)** — range: [1, 5]
  - Ordinal score from 1 to 5 assigned by an LLM judge (GPT-4o) or human experts, evaluating how well the agent's response and workflow address the research question and ground truth.
- `Accuracy [%]` — range: [0, 100]
  - Percentage of correctly answered multiple-choice questions (MCQs).
- `Recall [%]` — range: [0, 100]
  - Percentage of relevant items or correct components retrieved/identified by the agent out of all relevant items in the ground truth.
- `Precision [%]` — range: [0, 100]
  - Percentage of items identified by the agent that are actually relevant or correct according to the ground truth.

## Input / output format

**Input**: Experimental single-cell data (datasets < 750 MB) paired with an open-ended research question (OEQ) or multiple-choice question (MCQ).

**Output**: Autonomous multi-step workflow including code execution, data analysis, and a final textual answer or hypothesis addressing the research question.

## Scoring recipe

```python
if question_type == 'OEQ':
    score = llm_judge.evaluate(response, ground_truth, scale=[1,5])
    return score
else:
    predictions = extract_mcq_answer(response)
    gold = get_mcq_ground_truth(question)
    acc = (predictions == gold).mean() * 100
    recall = tp / (tp + fn) * 100
    precision = tp / (tp + fp) * 100
    return acc, recall, precision
```

## Common pitfalls

- Agents may crash or exceed time/API limits on large datasets; the benchmark restricts evaluation to the 'Lite' subset (<750 MB) for fair comparison.
- Correctness scores are ordinal and subject to LLM-judge bias; the paper validates alignment with human experts (Spearman ρ=0.90-0.93) but notes stochasticity from critic modules.
- Task prompts and agent configurations (e.g., step limits, critic placement) significantly impact results and must be standardized across runs.

## Evidence (verbatim from paper)

> We benchmark three state-of-the-art agents for single-cell biology on the sc-HeurekaBench: Biomni*(huang2025biomni)*, CellVoyager*(alber2025cellvoyager)*, and BixBench-Agent*(mitchener2025bixbench)*. ... We report results on questions related to datasets smaller than 750 MB, which we term sc-HeurekaBench-Lite. This subset contains 22 out of 50 OEQs and 18 out of 50 MCQs on which all agents could run. ... Correctness [1-5] | Accuracy [%] | Recall [%] | Precision [%]

## Citation

```bibtex
@misc{panigrahi2026heurekabench,
  title={HeurekaBench: A Benchmarking Framework for AI Co-scientist},
  author={Panigrahi et al. (2026)},
  year={2026},
  note={arXiv:2601.01678}
}
```

- arXiv: 2601.01678