Biopulse Qa Eval

---
name: biopulse-qa-eval
description: This benchmark evaluates large language models on biomedical question-answering, specifically probing their factuality, robustness to linguistic variations (paraphrasing and typos), and susceptibility to demographic bias (age and gender). It distinguishes between extractive and abstractive reasoning capabilities using expert-verified QA pairs from clinical documents. Use when the user wants to benchmark on BioPulse-QA, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.12632
  bibtex_key: [REDACTED]
  confidence: high
---

# biopulse-qa-eval

> BioPulse-QA: A Dynamic Biomedical Question-Answering Benchmark for Evaluating Factuality, Robustness, and Bias in Large Language Models — Bhattarai et al. (2026) (arXiv:2601.12632, 2026)

## What this evaluates

This benchmark evaluates large language models on biomedical question-answering, specifically probing their factuality, robustness to linguistic variations (paraphrasing and typos), and susceptibility to demographic bias (age and gender). It distinguishes between extractive and abstractive reasoning capabilities using expert-verified QA pairs from clinical documents.

## Datasets

- **BioPulse-QA** — total 2280; splits: test (2280)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Token-level F1 score (harmonic mean of precision and recall). The paper also reports a 'Relaxed F1' variant that tolerates minor formatting or whitespace differences while requiring exact match of core factual content.

## Input / output format

**Input**: A context snippet from a biomedical document (drug label, clinical trial protocol, or clinical guideline) followed by a question.

**Output**: Free-text answer string (either an exact span from the text for extractive tasks or a generated sentence for abstractive tasks).

## Scoring recipe

```python
def compute_f1(pred, gold):
    pred_tokens = set(normalize(pred))
    gold_tokens = set(normalize(gold))
    intersection = pred_tokens & gold_tokens
    precision = len(intersection) / len(pred_tokens) if pred_tokens else 0
    recall = len(intersection) / len(gold_tokens) if gold_tokens else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- High performance on abstractive QA does not guarantee accuracy on extractive QA; evaluating both is necessary to avoid masking weaknesses.
- Automated metrics like Relaxed F1 can overestimate correctness; manual error analysis is required to catch factual incorrectness and incompleteness.
- Robustness testing must include semantic paraphrasing, as models show high resilience to surface-level typos but significant sensitivity to rewording.

## Evidence (verbatim from paper)

> For abstractive QA, GPT-o1 achieved high F1 performance on drug labels, (0.98), and Gemini-2.0-Flash achieved high performance on clinical trials (0.65) and clinical guidelines (0.93) (Figure 4). Llama-3.1-8B underperformed with a low F1 score on drug labels (0.85), clinical trials (0.62), and clinical guidelines (0.69).

## Citation

```bibtex
@misc{bhattarai2026biopulseqa,
  title={BioPulse-QA: A Dynamic Biomedical Question-Answering Benchmark for Evaluating Factuality, Robustness, and Bias in Large Language Models},
  author={Bhattarai et al. (2026)},
  year={2026},
  note={arXiv:2601.12632}
}
```

- arXiv: 2601.12632