Bioinstruct Eval

---
name: bioinstruct-eval
description: Evaluates large language models on biomedical natural language processing tasks, including multiple-choice question answering, natural language inference, clinical information extraction, and text generation. It probes the model's ability to follow domain-specific instructions, extract precise medical entities, and generate coherent clinical notes or answers. Use when the user wants to benchmark on MedQA-USMLE, MedMCQA, PubmedQA, BioASQ MCQA, MedNLI, Medication Status Extraction, Coreference Resolution, Conv2note, ICliniq, MediQA-Task A, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.19975
  bibtex_key: tran2023bioinstruct
  confidence: high
---

# bioinstruct-eval

> BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing — Tran et al. (2023) (arXiv:2310.19975, 2023)

## What this evaluates

Evaluates large language models on biomedical natural language processing tasks, including multiple-choice question answering, natural language inference, clinical information extraction, and text generation. It probes the model's ability to follow domain-specific instructions, extract precise medical entities, and generate coherent clinical notes or answers.

## Datasets

- **MedQA-USMLE** — total ?; splits: test (-1)
- **MedMCQA** — total ?; splits: test (-1)
- **PubmedQA** — total ?; splits: test (-1)
- **BioASQ MCQA** — total ?; splits: test (-1)
- **MedNLI** — total ?; splits: test (-1)
- **Medication Status Extraction** — total ?; splits: test (-1)
- **Coreference Resolution** — total ?; splits: test (-1)
- **Conv2note** — total ?; splits: test (-1)
- **ICliniq** — total ?; splits: test (-1)
- **MediQA-Task A** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly predicted multiple-choice answers or NLI labels.
- `Precision/Recall/F1` — range: [0, 1]
  - Standard information extraction metrics computed over extracted clinical entities and their attributes.
- `Conditional ACC` — range: [0, 1]
  - Accuracy of medication status classification computed only on instances where the target medication was correctly extracted.
- `GPT-4 Coherence/Completeness/Naturalness` — range: [1, 5]
  - LLM-as-a-judge scores (typically 1-5 scale) generated by prompting GPT-4 to evaluate the quality of generated clinical notes or answers.
- `Concept_F1` — range: [0, 1]
  - F1 score computed over biomedical concept matches between generated text and reference.
- `BertScore_F1` — range: [0, 1]
  - F1 score based on contextual embeddings from BERT, measuring semantic similarity between generated and reference text.
- `Bleurt` — range: [0, 1]
  - Learned evaluation metric using BERT-based representations to score semantic similarity.

## Input / output format

**Input**: Instruction prompt containing a biomedical question, clinical context, or task description (e.g., doctor-patient dialogue, medical text).

**Output**: Model-generated text response, typically a multiple-choice letter, short answer, extracted entities, or a clinical note.

## Scoring recipe

```python
def score(predictions, gold, task_type):
    if task_type in ['QA', 'NLI']:
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'IE':
        # Compute Precision, Recall, F1 on extracted entities
        # Compute Conditional ACC: accuracy of status classification given correct extraction
        pass
    elif task_type == 'Generation':
        # GPT-4 scores (1-5) for Coherence, Completeness, Naturalness via API
        # Concept_F1, BertScore_F1, Bleurt via standard libraries
        pass
    return metric_value
```

## Common pitfalls

- GPT-4 evaluation metrics (Coherence, Completeness, Naturalness) are subjective and highly sensitive to the prompt template, temperature, and model version used for scoring.
- Conditional Accuracy for medication status extraction penalizes models for incorrect entity extraction before evaluating classification, conflating two distinct sub-tasks and potentially masking classification capability.
- Multi-task instruction tuning does not uniformly improve all task types; IE tasks often plateau or degrade when combined with generative tasks, contrary to the expectation that more tasks always help.

## Evidence (verbatim from paper)

> Table 1 illustrates results from several QA benchmarks in the biomedical domain, including MedQA-USMLE, MedMCQA, PubmedQA, and BioASQ MCQA. In the MedQA-USMLE, while the base LLaMA 1 7B secured 27.1, its instructed version (LLaMA 1 7B Instruct) rose to 31.58. This positive trend extends to tasks such as MedMCQA... Table 2 showcases the performance of various models on the clinical information extraction task, specifically focused on medication status... Table 3 encapsulates the performances of various models on the Conv2note, a challenging task aimed at converting doctor-patient conversations to clinical notes. Evidently, instruction tuning models, such as LLaMA 1 7B Instruct, consistently surpass their original versions, particularly in GPT-4 measures like Coherence, Completeness, and Naturalness. Specifically, LLaMA 1 7B Instruct sees a marked enhancement in Coherence, moving from 2.67 to 4.49. Furthermore, the instruction tuning also exhibits its strength in grasping medical concepts with LLaMA 7B Instruct attaining a Concept_F1 score of 31.51, a vast improvement over LLaMA 1 7B's 16.9.

## Citation

```bibtex
@misc{tran2023bioinstruct,
  title={BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing},
  author={Tran et al. (2023)},
  year={2023},
  note={arXiv:2310.19975}
}
```

- arXiv: 2310.19975