Biosage Scientific Eval

---
name: biosage-scientific-eval
description: Evaluates a compound AI architecture's ability to retrieve, synthesize, and reason across cross-disciplinary scientific knowledge. It probes performance on established single-domain science benchmarks and a novel benchmark specifically designed for bio-AI cross-domain synthesis and reasoning. Use when the user wants to benchmark on LitQA2, GPQA, WMDP, HLE-Bio, BioSage Cross-Disciplinary Benchmark, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.18298
  bibtex_key: [REDACTED]
  confidence: high
---

# biosage-scientific-eval

> Cross-Disciplinary Knowledge Retrieval and Synthesis: A Compound AI Architecture for Scientific Discovery — Volkova et al. (2025) (arXiv:2511.18298, 2025)

## What this evaluates

Evaluates a compound AI architecture's ability to retrieve, synthesize, and reason across cross-disciplinary scientific knowledge. It probes performance on established single-domain science benchmarks and a novel benchmark specifically designed for bio-AI cross-domain synthesis and reasoning.

## Datasets

- **LitQA2** — total ?; splits: test (-1)
- **GPQA** — total ?; splits: test (-1)
- **WMDP** — total ?; splits: test (-1)
- **HLE-Bio** — total ?; splits: test (-1)
- **BioSage Cross-Disciplinary Benchmark** — total 116; splits: test (116)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered multiple-choice questions out of the total number of questions in the benchmark.
- `precision` — range: [0, 1]
  - Proportion of relevant retrieved or synthesized items out of all items retrieved or generated by the system. Exact calculation thresholds are not specified in the text.

## Input / output format

**Input**: Multiple-choice scientific questions requiring cross-disciplinary reasoning. Prompts utilize a zero-shot approach with hybrid chunking and step-by-step reasoning instructions.

**Output**: Model-generated answer selection and/or detailed reasoning components explaining the choice.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    accuracy = correct / len(golds)
    # Precision calculation details are not provided in the text
    return {'accuracy': accuracy, 'precision': None}
```

## Common pitfalls

- The paper does not specify exact prompt templates, temperature settings, or chunking parameters used during evaluation.
- Precision is reported as a metric but lacks a defined formula, relevance threshold, or calculation method.
- The novel benchmark was synthetically generated using GPT-o1, which may introduce distribution shifts or biases compared to human-annotated datasets.

## Evidence (verbatim from paper)

> Benchmarks include LitQA2, GPQA, WMDP, and HLE-Bio. Each configuration was evaluated on identical test sets to enable direct performance comparisons, with metrics including accuracy and precision.

## Citation

```bibtex
@misc{volkova2025biosage,
  title={Cross-Disciplinary Knowledge Retrieval and Synthesis: A Compound AI Architecture for Scientific Discovery},
  author={Volkova et al. (2025)},
  year={2025},
  note={arXiv:2511.18298}
}
```

- arXiv: 2511.18298