Patientsim Eval

---
name: patientsim-eval
description: Evaluates the persona fidelity, factual accuracy, and clinical plausibility of an LLM-based patient simulator in doctor-patient dialogues. It measures how well the model adheres to assigned patient profiles, maintains factual consistency, and handles out-of-profile questions plausibly. Use when the user wants to benchmark on PatientSim Profiles, or asks about evaluating this task. Reports Entail (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.17818
  bibtex_key: kyung2025patientsim
  confidence: high
---

# patientsim-eval

> PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions — Kyung et al. (2025) (arXiv:2505.17818, 2025)

## What this evaluates

Evaluates the persona fidelity, factual accuracy, and clinical plausibility of an LLM-based patient simulator in doctor-patient dialogues. It measures how well the model adheres to assigned patient profiles, maintains factual consistency, and handles out-of-profile questions plausibly.

## Datasets

- **PatientSim Profiles** — total 170; splits: persona_eval (108), factuality_eval (52)

## Metrics

- `Entail (%)` **(primary)** — range: [0, 1]
  - Percentage of supported sentences (those related to at least one profile item) that are classified as entailment by an NLI evaluator. Calculated as the sum of entailment indicators for info-type sentences divided by the total number of info-type sentences.
- `Plausibility` — range: [1, 4]
  - 4-point scale rating (1-4) assigned to unsupported sentences (those containing information not in the profile) to measure clinical plausibility.
- `ICov` — range: [0, 1]
  - Proportion of profile item categories that appear in both the original profile and the LLM-derived profile from the dialogue. Calculated as the average over profiles of the ratio of overlapping items to total items.
- `ICon` — range: [1, 4]
  - Semantic similarity score between original and derived profile items for overlapping categories, rated on a 4-point scale by an LLM scorer.
- `Persona Fidelity Score` — range: [1, 4]
  - Average score across five criteria (Personality, Language, Recall, Confused, Realism) rated on a 4-point scale by human or LLM evaluators.

## Input / output format

**Input**: For persona evaluation: explicit persona descriptions and generated dialogues. For factuality/plausibility: dialogue history, current sentence, and the full patient profile containing predefined items. Evaluators receive step-specific instructions and rubrics.

**Output**: Persona evaluation: 4-point scores per criterion. Factuality: sentence type classification, binary relation vector to profile items, and NLI label (entailment/contradiction/neutral). Plausibility: 4-point plausibility score. Dialogue-level: derived profile items and semantic similarity scores.

## Scoring recipe

```python
def compute_entail_percent(utterances, profile):
    info_sents = [s for s in utterances if classify_type(s) == 'info']
    supported = [s for s in info_sents if any(relate(s, item) for item in profile)]
    entail_count = sum(1 for s in supported if nli_label(s, profile) == 'entailment')
    return entail_count / len(supported) if supported else 0

def compute_plausibility(utterances, profile):
    unsupported = [s for s in utterances if not relate(s, profile) or all(nli_label(s, profile) == 'neutral')]
    scores = [score_plausibility(s) for s in unsupported]  # 1-4 scale
    return sum(scores) / len(scores) if unsupported else 0

def compute_icov(original_profile, derived_profile):
    overlap = sum(1 for j in range(K) if original_profile[j] and derived_profile[j])
    return overlap / K

def compute_icon(original_profile, derived_profile):
    overlap_items = [j for j in range(K) if original_profile[j] and derived_profile[j]]
    if not overlap_items: return 0
    return sum(score_similarity(original_profile[j], derived_profile[j]) for j in overlap_items) / len(overlap_items)
```

## Common pitfalls

- Confusing supported vs. unsupported sentences: supported sentences are evaluated for factual entailment/contradiction, while unsupported ones are evaluated for clinical plausibility.
- Evaluator bias: The same LLM (Gemini-2.5-Flash) is used as both the doctor role in some setups and the primary scorer, which may inflate agreement scores.
- Dialogue-level metrics depend on an LLM profile extractor; extraction errors directly propagate to ICov and ICon scores.

## Evidence (verbatim from paper)

> The final factual accuracy of the sentence $s^{i}_{tm}$ is represented by Entail (%), calculated as follows, ... which reflects the percentage of supported sentences that are factually accurate.

## Citation

```bibtex
@misc{kyung2025patientsim,
  title={PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions},
  author={Kyung et al. (2025)},
  year={2025},
  note={arXiv:2505.17818}
}
```

- arXiv: 2505.17818