L2d Clinical Eval

---
name: l2d-clinical-eval
description: This evaluation probes the ability of an adaptive AI-to-AI deferral framework to selectively route clinical text classification tasks between domain-adapted BERT models and LLMs based on uncertainty signals. It measures whether intelligent routing improves classification accuracy while minimizing expensive LLM usage across binary and multi-class clinical NLP tasks. Use when the user wants to benchmark on ADE Corpus V2, MIMIC-IV Treatment Outcomes, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13285
  bibtex_key: [REDACTED]
  confidence: high
---

# l2d-clinical-eval

> L2D-Clinical: Learning to Defer for Adaptive Model Selection in Clinical Text Classification — Kondadadi et al. (2026) (arXiv:2604.13285, 2026)

## What this evaluates

This evaluation probes the ability of an adaptive AI-to-AI deferral framework to selectively route clinical text classification tasks between domain-adapted BERT models and LLMs based on uncertainty signals. It measures whether intelligent routing improves classification accuracy while minimizing expensive LLM usage across binary and multi-class clinical NLP tasks.

## Datasets

- **ADE Corpus V2** — total 23516; splits: train (-1), val (-1), test (-1); HF `ade-benchmark-corpus/ade_corpus_v2`
- **MIMIC-IV Treatment Outcomes** — total 2782; splits: train (2225), val (278), test (279)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Computed on the positive class for binary ADE detection and macro-averaged across all classes for 3-class treatment outcomes.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified instances out of the total number of instances.
- `LLM%` — range: percent
  - Percentage of total test instances routed to the LLM by the deferral model instead of the BERT model.

## Input / output format

**Input**: Clinical text instances: English sentences from PubMed medical case reports for ADE detection, and drug-disease-outcome triplets extracted from MIMIC-IV discharge summaries for treatment outcome classification.

**Output**: Binary label (1 for ADE present, 0 for absent) or one of three categorical labels (EFFECTIVE, ADVERSE, NEUTRAL), plus a deferral decision indicating whether the instance is classified by the BERT model or routed to the LLM.

## Scoring recipe

```python
def compute_metrics(predictions, gold, labels):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = correct / len(gold)
    f1_scores = []
    for label in labels:
        tp = sum(1 for p, g in zip(predictions, gold) if p == label and g == label)
        fp = sum(1 for p, g in zip(predictions, gold) if p == label and g != label)
        fn = sum(1 for p, g in zip(predictions, gold) if p != label and g == label)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    llm_deferred = sum(1 for p in predictions if p == 'LLM')
    llm_pct = llm_deferred / len(predictions) * 100
    return {'accuracy': accuracy, 'f1': macro_f1, 'llm_pct': llm_pct}
```

## Common pitfalls

- Small test set sizes (500 for ADE, 279 for MIMIC-IV) mean reported F1 improvements <0.01 may not be statistically significant without bootstrap confidence intervals.
- LLM evaluation is restricted to a 500-instance subset of the ADE test set due to API costs, requiring careful fair comparison with BERT which is evaluated on the full test set (or the same 500 subset).
- Ground truth for MIMIC-IV is derived from multi-LLM consensus labeling rather than human annotation, which may introduce systematic biases or differ from clinical expert judgments.

## Evidence (verbatim from paper)

> We report F1 score (harmonic mean of precision and recall) as our primary metric, computed on the positive class for binary ADE detection and macro-averaged across classes for 3-class treatment outcomes. We also report accuracy and the percentage of instances deferred to the LLM (LLM%). For ADE detection, we use 5-fold cross-validation to obtain out-of-fold deferral probabilities, reducing overfitting risk. Given the moderate test set sizes (500 and 279 samples), we acknowledge that small F1 differences (<0.01) may not be statistically significant; our reported improvements of +1.7 points (ADE) and +9.3 points (MIMIC) represent meaningful gains.

## Citation

```bibtex
@misc{kondadadi2026l2dclinical,
  title={L2D-Clinical: Learning to Defer for Adaptive Model Selection in Clinical Text Classification},
  author={Kondadadi et al. (2026)},
  year={2026},
  note={arXiv:2604.13285}
}
```

- arXiv: 2604.13285