Biomedqa Eval

---
name: biomedqa-eval
description: Probes an AI agent's ability to answer pharmacology questions by querying federated biomedical knowledge graphs. It evaluates three access methods—direct MCP tools, text-to-Cypher generation, and standalone LLM reasoning—to measure factual accuracy and query efficiency. Use when the user wants to benchmark on BiomedQA, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15080
  bibtex_key: mandarapu2026samyama
  confidence: high
---

# biomedqa-eval

> Open Biomedical Knowledge Graphs at Scale: Construction, Federation, and AI Agent Access with Samyama Graph Database — Mandarapu et al. (2026) (arXiv:2603.15080, 2026)

## What this evaluates

Probes an AI agent's ability to answer pharmacology questions by querying federated biomedical knowledge graphs. It evaluates three access methods—direct MCP tools, text-to-Cypher generation, and standalone LLM reasoning—to measure factual accuracy and query efficiency.

## Datasets

- **BiomedQA** — total 40; splits: test (40); repo https://github.com/samyama-ai/biomedqa

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total 40 pharmacology questions across 7 categories.

## Input / output format

**Input**: 40 pharmacology questions spanning 7 categories, requiring answers derived from federated biomedical knowledge graphs (Pathways, Drug Interactions, Clinical Trials).

**Output**: Natural language answer or executed query result, depending on the evaluation approach (MCP tools, text-to-Cypher, or standalone LLM).

## Scoring recipe

```python
correct = 0
for question, prediction, gold in zip(questions, predictions, gold_answers):
    if prediction == gold:
        correct += 1
accuracy = (correct / len(questions)) * 100
```

## Common pitfalls

- Schema hallucinations in text-to-Cypher approaches (e.g., non-existent edge traversals) can cause failures unrelated to factual knowledge.
- Token counting conventions differ across approaches; the NLQ endpoint handles LLM calls server-side, resulting in 0 reported tokens for that method.
- Latency varies drastically between single-KG lookups (~80-100 ms) and multi-hop cross-KG joins (3-4 s), so benchmarking conditions must be explicitly stated.

## Evidence (verbatim from paper)

> We introduce BiomedQA, a benchmark of 40 pharmacology questions across 7 categories over the three federated KGs. We compare three approaches: domain-specific MCP tools (parameterized Cypher templates), text-to-Cypher via the schema-aware NLQ endpoint (GPT-4o with full schema system prompt and few-shot examples), and standalone GPT-4o (no database access). Table 10: BiomedQA results (40 questions, 7.9M nodes, 3 federated KGs).

| Approach | Accuracy | Avg Latency | Avg Tokens |

## Citation

```bibtex
@misc{mandarapu2026samyama,
  title={Open Biomedical Knowledge Graphs at Scale: Construction, Federation, and AI Agent Access with Samyama Graph Database},
  author={Mandarapu et al. (2026)},
  year={2026},
  note={arXiv:2603.15080}
}
```

- arXiv: 2603.15080