Ukb Disease Risk Eval

---
name: ukb-disease-risk-eval
description: Evaluates the ability of multimodal LLMs to predict binary disease risk from individual-specific clinical data, including tabular features and time-series spirograms. It tests how well serialized text and cross-modal embeddings integrate to produce accurate risk scores for conditions like asthma and diabetes. Use when the user wants to benchmark on UK Biobank, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.09018
  bibtex_key: belyaeva2023multimodal
  confidence: high
---

# ukb-disease-risk-eval

> Multimodal LLMs for health grounded in individual-specific data — Belyaeva et al. (2023) (arXiv:2307.09018, 2023)

## What this evaluates

Evaluates the ability of multimodal LLMs to predict binary disease risk from individual-specific clinical data, including tabular features and time-series spirograms. It tests how well serialized text and cross-modal embeddings integrate to produce accurate risk scores for conditions like asthma and diabetes.

## Datasets

- **UK Biobank** — total ?; splits: train (-1), val (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `AUPRC` — range: [0, 1]
  - Area Under the Precision-Recall Curve, measuring performance on the positive class across classification thresholds.

## Input / output format

**Input**: Serialized tabular data formatted as JSON-like key-value pairs combined with a base prompt (e.g., 'Predict if a patient has the condition or not. bmi: {val}. age: {val}.'). For multimodal HeLM, non-text modalities (spirograms, tabular) are projected into the LLM's token embedding space via modality-specific encoders and concatenated with text tokens.

**Output**: The model generates a completion. The evaluation computes the log-likelihood of the completion being “yes}” or “no}” to derive a risk score.

## Scoring recipe

```python
# Per instance
ll_yes = model.log_likelihood(prompt + "yes}")
ll_no = model.log_likelihood(prompt + "no}")
risk_score = ll_yes  # or ll_yes - ll_no

# Aggregate across dataset
auroc = compute_auroc(risk_scores, true_labels)
auprc = compute_auprc(risk_scores, true_labels)
```

## Common pitfalls

- Risk scores are derived from the log-likelihood of specific token completions ('yes}' or 'no}') rather than standard classification probabilities or logits.
- The evaluation is restricted to a European ancestry subset of UK Biobank, which may introduce population bias and limit cross-ancestry generalizability.
- Spirogram inputs are preprocessed into fixed-length 1D flow-volume curves, meaning the model does not evaluate raw time-series directly.

## Evidence (verbatim from paper)

> We then compute the log-likelihood of the sentence being completed with “yes}” or “no}”. This log-likelihood serves as a risk score and can be evaluated using metrics such as AUROC and AUPRC to assess discriminatory power.

## Citation

```bibtex
@misc{belyaeva2023multimodal,
  title={Multimodal LLMs for health grounded in individual-specific data},
  author={Belyaeva et al. (2023)},
  year={2023},
  note={arXiv:2307.09018}
}
```

- arXiv: 2307.09018