Deepimagine Clinical Trial Eval

---
name: deepimagine-clinical-trial-eval
description: Evaluates a model's ability to perform counterfactual reasoning in biomedical settings by predicting clinical trial outcomes under perturbations of either the outcome measure or the study arm, using similarity-based retrieval to construct counterfactual pairs. Use when the user wants to benchmark on CT open evaluation sample, or asks about evaluating this task. Reports clinical trial outcome prediction.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.23054
  bibtex_key: [REDACTED]
  confidence: medium
---

# deepimagine-clinical-trial-eval

> DeepImagine: Learning Biomedical Reasoning via Successive Counterfactual Imagining — Zheng et al. (2026) (arXiv:2604.23054, 2026)

## What this evaluates

Evaluates a model's ability to perform counterfactual reasoning in biomedical settings by predicting clinical trial outcomes under perturbations of either the outcome measure or the study arm, using similarity-based retrieval to construct counterfactual pairs.

## Datasets

- **CT open evaluation sample** — total 500; splits: test (500)

## Metrics

- `clinical trial outcome prediction` **(primary)** — range: [0, 1]
  - Accuracy or error rate in predicting the target clinical trial outcome result given a counterfactual perturbation (exact formula not specified in the provided section).

## Input / output format

**Input**: For outcome-measure perturbation: reported result of the most similar alternative outcome measure within the same trial. For study-arm perturbation: reported result of a different study arm measured under the same outcome measure within the same trial.

**Output**: Predicted result for the target outcome measure or fixed study arm.

## Scoring recipe

```python
def score(predictions, gold):
    # Exact metric not specified; assumes classification/regression accuracy
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Evaluation data is sampled from the training split of the source dataset (CT open), risking data leakage if the base model was pre-trained or fine-tuned on it.
- Counterfactual pairs are constructed via embedding similarity (NV-embed), which may introduce retrieval bias or semantic drift not controlled for.
- Two distinct perturbation types (outcome-measure vs. study-arm) are evaluated on the same 500-question pool but reported separately, making aggregate comparison difficult.

## Evidence (verbatim from paper)

> We evaluate our approach by its downstream utility for clinical trial outcome prediction. From the training split of [[28]], we draw 250 random questions from each of the Superiority and Comparative Effect classes... A evaluated model is provided with the reported result of the nearest outcome measure and asked to predict the target outcome measure’s result under the same study arm.

## Citation

```bibtex
@misc{zheng2026deepimagine,
  title={DeepImagine: Learning Biomedical Reasoning via Successive Counterfactual Imagining},
  author={Zheng et al. (2026)},
  year={2026},
  note={arXiv:2604.23054}
}
```

- arXiv: 2604.23054