Biological Mllm Merging Eval

---
name: biological-mllm-merging-eval
description: Evaluates the ability of merged multimodal large language models to perform cross-modal biological reasoning tasks, specifically predicting molecular interactions with proteins/cells and predicting enzyme functionality. It probes whether embedding-space-aware merging preserves modality-specific expertise better than parameter-space heuristics or fine-tuning. Use when the user wants to benchmark on Biological MLLM Interaction & Functionality Benchmarks, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.14405
  bibtex_key: lee2026esmerging
  confidence: high
---

# biological-mllm-merging-eval

> ES-Merging: Biological MLLM Merging via Embedding Space Signals — Lee et al. (2026) (arXiv:2603.14405, 2026)

## What this evaluates

Evaluates the ability of merged multimodal large language models to perform cross-modal biological reasoning tasks, specifically predicting molecular interactions with proteins/cells and predicting enzyme functionality. It probes whether embedding-space-aware merging preserves modality-specific expertise better than parameter-space heuristics or fine-tuning.

## Datasets

- **Biological MLLM Interaction & Functionality Benchmarks** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted instances out of the total number of instances.
- `Macro-F1` — range: [0, 1]
  - Unweighted mean of the F1 scores calculated for each class independently, where F1 is the harmonic mean of precision and recall.

## Input / output format

**Input**: Multimodal inputs (molecule, protein, or cell representations) paired with task-specific few-shot in-context examples and a standardized instruction template. Unsupported modalities in baselines are converted to textual inputs.

**Output**: Classification predictions (e.g., interaction yes/no, inhibition/substrate specificity, or enzyme functionality classes).

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    acc = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    classes = sorted(set(gold_labels))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    return acc, macro_f1
```

## Common pitfalls

- Unsupported modalities are handled differently across methods (textual conversion for baselines vs. native multimodal for ES-Merging), which can skew comparisons if not standardized.
- Task-specific fine-tuning baselines may overfit to specific datasets, leading to instability or degraded generalization on unseen cross-modal pairs.
- Evaluation relies on few-shot in-context learning, so prompt sensitivity and example selection can significantly impact reported scores.

## Evidence (verbatim from paper)

> We consider two instance-varying cross-modal interaction settings: molecule-protein interaction and molecule-cell interaction. For molecule-protein interaction, the task is to predict whether a given molecule interacts with a given protein, including BindingDB, BioSNAP, and Human. For molecule-cell interaction, the task is to predict the effect of a molecule on a given cell, including DrugComb and GDSC2. ... To this end, we evaluate on CYP enzyme prediction. ... We report accuracy and macro-F1 across each subset.

## Citation

```bibtex
@misc{lee2026esmerging,
  title={ES-Merging: Biological MLLM Merging via Embedding Space Signals},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2603.14405}
}
```

- arXiv: 2603.14405