Drugpc Eval

---
name: drugpc-eval
description: Probes multi-step therapeutic reasoning and tool-use for drug-related questions, including interactions, contraindications, and patient-specific treatment strategies. It tests the model's ability to dynamically select biomedical tools, retrieve verified knowledge, and generate evidence-grounded answers. Use when the user wants to benchmark on DrugPC, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.10970
  bibtex_key: gao2025txagent
  confidence: high
---

# drugpc-eval

> TxAgent: An AI Agent for Therapeutic Reasoning Across a Universe of Tools — Gao et al. (2025) (arXiv:2503.10970, 2025)

## What this evaluates

Probes multi-step therapeutic reasoning and tool-use for drug-related questions, including interactions, contraindications, and patient-specific treatment strategies. It tests the model's ability to dynamically select biomedical tools, retrieve verified knowledge, and generate evidence-grounded answers.

## Datasets

- **DrugPC** — total 3168; splits: test (3168)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered questions. Evaluated in two settings: multiple-choice (selecting from 4–5 predefined options) and open-ended (generating a free-text response, then mapping it to the correct option).

## Input / output format

**Input**: Therapeutic questions about drugs, diseases, or patient populations. In multiple-choice mode: question plus 4–5 answer options. In open-ended mode: question only.

**Output**: A multi-step reasoning trace (thought process, function calls, tool feedback) followed by a final answer. For evaluation, the final answer is matched to the correct option.

## Scoring recipe

```python
correct = 0
for q, gold_opt in dataset:
    if setting == 'multiple_choice':
        pred = model.generate(q, options)
    else:
        pred_text = model.generate(q)
        pred = extract_option_from_text(pred_text, options)
    if pred == gold_opt:
        correct += 1
return correct / len(dataset)
```

## Common pitfalls

- Data leakage: Pretraining data often contains drugs approved before 2024, so the benchmark restricts evaluation to 2024 FDA-approved drugs to ensure fair testing of real-time tool use.
- Open-ended vs. multiple-choice gap: Models heavily rely on answer choices in MC format; switching to open-ended causes significant accuracy drops for baselines, revealing true reasoning capability.
- Tool execution verification: Accuracy alone does not capture whether the model actually invoked the correct tools or hallucinated the answer; reasoning traces must be inspected for valid tool calls.

## Evidence (verbatim from paper)

> We construct the DrugPC (Drug Prescribing Card) benchmark to evaluate TxAgent’s performance in drug reasoning. DrugPC includes 3,168 questions spanning 11 tasks: drug overview, ingredients, warnings and safety, dependence and abuse, dosage and administration, use in specific populations, pharmacology, clinical information, nonclinical toxicology, patient-focused information, and storage and supply. We evaluate models in two settings: multiple-choice, where the model selects the correct answer from given options, and open-ended, where the model generates responses without predefined choices. In the multiple-choice setting, TxAgent achieves 93.8% accuracy... In the open-ended setting, TxAgent maintains 92.1% accuracy...

## Citation

```bibtex
@misc{gao2025txagent,
  title={TxAgent: An AI Agent for Therapeutic Reasoning Across a Universe of Tools},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2503.10970}
}
```

- arXiv: 2503.10970