Huatuo 26m Eval

---
name: huatuo-26m-eval
description: Evaluates Chinese medical question-answering capabilities through retrieval and generation tasks. It probes domain-specific knowledge retrieval from large pools and tests generative models on producing accurate, long-form medical answers. Use when the user wants to benchmark on Huatuo-26M, or asks about evaluating this task. Reports Recall@5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.01526
  bibtex_key: li2023huatuo26m
  confidence: high
---

# huatuo-26m-eval

> Huatuo-26M, a Large-scale Chinese Medical QA Dataset — Jianquan Li et al. (2023) (arXiv:2305.01526, 2023)

## What this evaluates

Evaluates Chinese medical question-answering capabilities through retrieval and generation tasks. It probes domain-specific knowledge retrieval from large pools and tests generative models on producing accurate, long-form medical answers.

## Datasets

- **Huatuo-26M** — total 26000000; splits: train (-1), test (-1); repo https://github.com/FreedomIntelligence/Huatuo-26M

## Metrics

- `Recall@5` **(primary)** — range: [0, 1]
  - Measures the percentage of queries for which the correct answer appears in the top 5 retrieved passages from the candidate pool.
- `MRR@10` — range: [0, 1]
  - Calculates the average of the inverse of the rank at which the first relevant document is retrieved, considering only the top 10 results.
- `BLEU-4` — range: [0, 1]
  - Computes the 4-gram overlap between the generated answer and the reference answer, with a brevity penalty.
- `ROUGE-L` — range: [0, 1]
  - Measures the longest common subsequence of words between the generated and reference answers.

## Input / output format

**Input**: A Chinese medical question.

**Output**: For retrieval: a ranked list of candidate answers. For generation: a text string representing the predicted medical answer.

## Scoring recipe

```python
def compute_recall_mrr(retrieved_lists, gold_answers, k=5):
    correct = 0
    mrr_sum = 0.0
    for preds, gold in zip(retrieved_lists, gold_answers):
        if gold in preds[:k]:
            correct += 1
        for rank, pred in enumerate(preds[:10], 1):
            if pred == gold:
                mrr_sum += 1.0 / rank
                break
    recall_k = correct / len(gold_answers)
    mrr_10 = mrr_sum / len(gold_answers)
    return recall_k, mrr_10
```

## Common pitfalls

- The retrieval candidate pool is extremely large (26M), causing inherently low recall scores even when top-ranked answers are informative.
- Generation metrics like BLEU/ROUGE heavily penalize long, medically accurate answers due to strict n-gram overlap requirements, underestimating model capability.
- Split ratios vary by data source (90/10 for encyclopedias/knowledge bases vs. 99/1 for consultant records), making cross-source comparisons sensitive to pool size and test set composition.

## Evidence (verbatim from paper)

> We use Recall@k and MRR@10 as evaluation indicators. Recall@k measures the percentage of top k retrieved passages that contain the answer. MRR@10 calculates the average of the inverse of the ranks at which the first relevant document was retrieved.

## Citation

```bibtex
@misc{li2023huatuo26m,
  title={Huatuo-26M, a Large-scale Chinese Medical QA Dataset},
  author={Jianquan Li et al. (2023)},
  year={2023},
  note={arXiv:2305.01526}
}
```

- arXiv: 2305.01526