Eva Immunology Benchmark Eval

---
name: eva-immunology-benchmark-eval
description: Evaluates a multimodal foundation model's ability to predict drug efficacy, classify disease endotypes, and align cross-species transcriptomic and histological data for immunology and inflammation research. Use when the user wants to benchmark on I&I benchmark, IBDome dataset, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.10168
  bibtex_key: bandasack2026eva
  confidence: high
---

# eva-immunology-benchmark-eval

> EVA: Towards a universal model of the immune system — Bandasack et al. (2026) (arXiv:2602.10168, 2026)

## What this evaluates

Evaluates a multimodal foundation model's ability to predict drug efficacy, classify disease endotypes, and align cross-species transcriptomic and histological data for immunology and inflammation research.

## Datasets

- **I&I benchmark** — total ?; splits: test (-1)
- **IBDome dataset** — total ?; splits: test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve; measures the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance.
- `accuracy` — range: [0, 1]
  - Proportion of correctly classified instances out of the total number of instances.

## Input / output format

**Input**: Patient-level transcriptomic data (bulk RNA-seq, microarray, pseudobulk scRNA-seq) or histology tile embeddings; for zero-shot tasks, inputs are perturbed patient transcriptomes simulating target up/downregulation.

**Output**: Continuous efficacy score between 0 and 1 for zero-shot tasks; discrete class labels (e.g., binary inflammation status, Montreal disease course categories, histopathological diagnosis) for classification tasks.

## Scoring recipe

```python
def score_zero_shot(patient_exprs, target, healthy_ref):
    scores = []
    for expr in patient_exprs:
        perturbed = simulate_perturbation(expr, target)
        shift = geometric_distance(perturbed, healthy_ref)
        scores.append(normalize_0_1(shift))
    median_score = median(scores)
    return median_score

# Rank drug-disease pairs by median_score
# Compute AUROC against ground_truth_efficacy
auc = compute_auroc(ranked_predictions, ground_truth_efficacy)
```

## Common pitfalls

- BulkRNABert decoder is not publicly available, preventing direct zero-shot efficacy evaluation.
- Drug-disease efficacy labels are incomplete due to limited clinical trial evidence, requiring careful handling of missing data.
- Performance is highly sensitive to the decision threshold (e.g., 0.5) used to convert continuous scores to binary predictions.

## Evidence (verbatim from paper)

> For each patient in a disease cohort, we simulated the perturbation of relevant drug targets and measured geometrically how the resulting transcriptomic state shifted relative to healthy reference tissue. Each patient received a score between 0 and 1, where higher values indicate greater alignment toward the healthy phenotype, suggesting potential therapeutic benefit. We then computed the median score across all patients for each drug-disease combination. Drug-disease pairs were ranked by their median alignment scores, and we computed AUROC to assess discrimination between efficacious and non-efficacious treatments.

## Citation

```bibtex
@misc{bandasack2026eva,
  title={EVA: Towards a universal model of the immune system},
  author={Bandasack et al. (2026)},
  year={2026},
  note={arXiv:2602.10168}
}
```

- arXiv: 2602.10168