Medical Dialogue Gen Eval

---
name: medical-dialogue-gen-eval
description: This evaluation probes a model's ability to generate clinically accurate, fluent, and context-aware physician responses in multi-turn medical dialogues. It assesses both automatic language quality and semantic relevance, alongside human-rated fluency, knowledge correctness, and overall satisfaction. Use when the user wants to benchmark on KaMed, MedDialog, MedDG, or asks about evaluating this task. Reports BLEU-2.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2105.06071
  bibtex_key: [REDACTED]
  confidence: high
---

# medical-dialogue-gen-eval

> Semi-Supervised Variational Reasoning for Medical Dialogue Generation — Dongdong Li et al. (2021) (arXiv:2105.06071, 2021)

## What this evaluates

This evaluation probes a model's ability to generate clinically accurate, fluent, and context-aware physician responses in multi-turn medical dialogues. It assesses both automatic language quality and semantic relevance, alongside human-rated fluency, knowledge correctness, and overall satisfaction.

## Datasets

- **KaMed** — total ?; splits: train (57754), valid (3000), test (3000)
- **MedDialog** — total ?; splits: train (32723), valid (3000), test (3000)
- **MedDG** — total ?; splits: train (14864), valid (2000), test (1000)

## Metrics

- `BLEU-2` **(primary)** — range: [0, 1]
  - Standard BLEU score computed over 2-grams, combining precision with a brevity penalty to penalize overly short generations.
- `ROUGE-2` — range: [0, 1]
  - Recall-oriented overlap of 2-grams between the generated response and the reference response.
- `Distinct-2` — range: [0, 1]
  - The proportion of distinct 2-grams in the generated responses, measuring lexical diversity.
- `Entity F1` — range: [0, 1]
  - Macro and micro average Precision, Recall, and F1 scores of predicted medical entities in the generated responses.
- `Human EQ` — range: [1, 5]
  - Overall quality rating on a 1-5 Likert scale provided by professional medical annotators, with model names masked.

## Input / output format

**Input**: Multi-turn dialogue context consisting of patient utterances, optionally augmented with medical knowledge graph entities and triplets.

**Output**: A single natural language physician response string.

## Scoring recipe

```python
def compute_rouge2(gold, pred):
    gold_tokens = tokenize(gold)
    pred_tokens = tokenize(pred)
    gold_ngrams = set(ngrams(gold_tokens, 2))
    pred_ngrams = set(ngrams(pred_tokens, 2))
    if not gold_ngrams:
        return 0.0
    overlap = len(gold_ngrams & pred_ngrams)
    return overlap / len(gold_ngrams)
```

## Common pitfalls

- Relying solely on BLEU/ROUGE scores, as the paper explicitly notes these have known shortcomings for dialogue generation quality.
- Failing to mask model names during human evaluation, which can introduce annotator bias and invalidate Flu/KC/EQ ratings.
- Ignoring the semi-supervised setup; KaMed and MedDialog lack annotations, so evaluation must account for models trained on unlabeled data.

## Evidence (verbatim from paper)

> To assess the language quality for the generated responses, we employ classical word-overlap based metrics, BLEU-2 (B@2) [42] and ROUGE-2 (R@2)[31], to measure performance. As shortcomings have been reported for using BLEU/ROUGE to measure dialogue generation [33], we also use Distinct-1 (D@1) and Distinct-2 (D@2) [27], where Distinct-n is defined as the proportion of distinct n-grams in generated responses.

## Citation

```bibtex
@misc{li2021semisupervised,
  title={Semi-Supervised Variational Reasoning for Medical Dialogue Generation},
  author={Dongdong Li et al. (2021)},
  year={2021},
  note={arXiv:2105.06071}
}
```

- arXiv: 2105.06071