Dutch Medical Dialogue Eval

---
name: dutch-medical-dialogue-eval
description: Evaluates the quality of synthetically generated Dutch medical dialogues across structural, lexical, and qualitative dimensions to assess conversational naturalness and domain-specific accuracy. Use when the user wants to benchmark on Synthetic Dutch Medical Dialogues, or asks about evaluating this task. Reports MSTTR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.09645
  bibtex_key: kuan2026generating
  confidence: high
---

# dutch-medical-dialogue-eval

> Generating High Quality Synthetic Data for Dutch Medical Conversations — Kuan et al. (2026) (arXiv:2604.09645, 2026)

## What this evaluates

Evaluates the quality of synthetically generated Dutch medical dialogues across structural, lexical, and qualitative dimensions to assess conversational naturalness and domain-specific accuracy.

## Datasets

- **Synthetic Dutch Medical Dialogues** — total 9; splits: test (9)

## Metrics

- `MSTTR` **(primary)** — range: [0, 1]
  - Moving-Segment Type-Token Ratio using overlapping sliding windows to measure local lexical variation. The paper notes MATTR was also computed but MSTTR was reported.
- `Alternation rate` — range: [0, 1]
  - Proportion of turns that alternate between doctor and patient roles, calculated as alternating transitions divided by total turns.
- `Role consistency` — range: [0, 1]
  - Keyword matching overlap score between role-specific lexicons (doctor/patient) and the generated dialogue text.
- `Topic coverage` — range: [0, 1]
  - Proportion of target clinical topics covered in the dialogue, measured via keyword matching against a predefined topic list.
- `Human qualitative score` — range: [1, 5]
  - Rater scores on a 5-point scale across five categories: Coherence, Consistency, Relevance, Clinical Use, and Fluency.

## Input / output format

**Input**: Prompt containing a clinical topic/scenario, two-shot examples of real Dutch medical conversations, and structural constraints (e.g., one-sentence-per-turn).

**Output**: Multi-turn Dutch dialogue text alternating between doctor and patient roles.

## Scoring recipe

```python
def evaluate_dialogue(dialogue_text, gold_topics, doctor_lexicon, patient_lexicon):
    turns = split_into_turns(dialogue_text)
    sentences = split_into_sentences(dialogue_text)
    words = tokenize(dialogue_text)
    
    # Structural metrics
    roles = detect_roles(turns)
    alternation_rate = sum(roles[i] != roles[i-1] for i in range(1, len(roles))) / (len(roles) - 1)
    asl = len(sentences) / len(turns)
    spt = len(sentences) / len(turns)
    
    # Lexical & Role metrics
    ttr = len(set(words)) / len(words)
    msttr = moving_average_ttr(words, window_size=50)
    role_consistency = (count_keyword_matches(turns[::2], doctor_lexicon) + count_keyword_matches(turns[1::2], patient_lexicon)) / (len(turns) * max_vocab)
    topic_coverage = count_keyword_matches(dialogue_text, gold_topics) / len(gold_topics)
    
    return {
        'alternation_rate': alternation_rate,
        'role_consistency': role_consistency,
        'asl': asl,
        'spt': spt,
        'topic_coverage': topic_coverage,
        'ttr': ttr,
        'msttr': msttr
    }
```

## Common pitfalls

- Automatic metrics like MSTTR and alternation rate correlate poorly with human ratings of conversational naturalness, capturing pattern regularity rather than semantic/pragmatic quality.
- Keyword-based role consistency and topic coverage fail to capture semantic correctness or contextual appropriateness of domain-specific vocabulary.
- Small sample size (n=9) and low inter-rater reliability (Krippendorff's α < 0.12) limit statistical generalizability and highlight rubric ambiguity.

## Evidence (verbatim from paper)

> Correlation analysis (Figure 12) confirmed weak alignment between automatic and human assessments: fluency and clinical use correlated moderately with MSTTR and role consistency, whereas relevance showed a negative relation (ρ = –0.31). Given the limited number of dialogues, these correlations should be interpreted with caution. These discrepancies highlight that numeric metrics capture pattern regularity but not semantic or pragmatic naturalness.

## Citation

```bibtex
@misc{kuan2026generating,
  title={Generating High Quality Synthetic Data for Dutch Medical Conversations},
  author={Kuan et al. (2026)},
  year={2026},
  note={arXiv:2604.09645}
}
```

- arXiv: 2604.09645