Liveclin Eval

---
name: liveclin-eval
description: LiveClin probes real-world clinical reasoning and longitudinal case management by evaluating models on dynamic, multimodal patient scenarios. It tests the ability to maintain context across sequential diagnostic, treatment, and follow-up questions while resisting data contamination from static training corpora. Use when the user wants to benchmark on LiveClin, or asks about evaluating this task. Reports Case Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.16747
  bibtex_key: wang2026liveclin
  confidence: high
---

# liveclin-eval

> LiveClin: A Live Clinical Benchmark without Leakage — Wang et al. (2026) (arXiv:2602.16747, 2026)

## What this evaluates

LiveClin probes real-world clinical reasoning and longitudinal case management by evaluating models on dynamic, multimodal patient scenarios. It tests the ability to maintain context across sequential diagnostic, treatment, and follow-up questions while resisting data contamination from static training corpora.

## Datasets

- **LiveClin** — total ?; splits: test (-1); repo https://github.com/AQ-MedAI/LiveClin

## Metrics

- `Case Accuracy` **(primary)** — range: percent
  - A case is deemed correct only if all of its sequential questions are answered correctly. It is calculated as the percentage of cases where every turn in the conversation matches the gold standard.

## Input / output format

**Input**: Conversational, zero-shot prompts where the full conversation history is maintained as context for each subsequent question. Cases are multimodal, spanning text, images, and tables across diagnosis, treatment, and long-term management stages.

**Output**: Sequential answers to multi-turn questions within a clinical case conversation.

## Scoring recipe

```python
def compute_case_accuracy(predictions, gold):
    # predictions, gold: lists of model/gold answers per sequential question in a case
    all_correct = all(p == g for p, g in zip(predictions, gold))
    return 1.0 if all_correct else 0.0
```

## Common pitfalls

- Maintaining full conversation history across sequential questions can cause context degradation or attention loss, especially in late-stage follow-up questions.
- Performance is highly modality-dependent; models excel at extracting structured data (e.g., Diagrams) but fail on complex inference tasks (e.g., Pathology, Biosignals).
- The zero-shot conversational setup does not allow for iterative refinement or feedback, penalizing models that typically rely on multi-turn clarification.

## Evidence (verbatim from paper)

> To faithfully simulate sequential clinical encounters, we employed a conversational, zero-shot evaluation protocol. The full conversation history is maintained as context for each subsequent question, forcing the model to continuously integrate new information. For reproducibility, we set temperature to 0 for most models, adopting official recommended configurations for those with specific reasoning modes. Our primary metric is Case Accuracy, a stringent measure where a case is deemed correct only if all of its sequential questions are answered correctly.

## Citation

```bibtex
@misc{wang2026liveclin,
  title={LiveClin: A Live Clinical Benchmark without Leakage},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.16747}
}
```

- arXiv: 2602.16747