Reactembed Eval

---
name: reactembed-eval
description: Evaluates a cross-domain representation learning framework for protein-molecule interactions by measuring prediction accuracy on regression and classification tasks across diverse biochemical benchmarks. Use when the user wants to benchmark on FreeSolv, CEP, BetaLactamase, Stability, BindingDB, PPIAffinity, BBBP, GO-CC, DrugBank, HumanPPI, YeastPPI, or asks about evaluating this task. Reports Root Mean Square Error (RMSE).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.18278
  bibtex_key: sicherman2025reactembed
  confidence: high
---

# reactembed-eval

> ReactEmbed: A Cross-Domain Framework for Protein-Molecule Representation Learning via Biochemical Reaction Networks — Sicherman et al. (2025) (arXiv:2501.18278, 2025)

## What this evaluates

Evaluates a cross-domain representation learning framework for protein-molecule interactions by measuring prediction accuracy on regression and classification tasks across diverse biochemical benchmarks.

## Datasets

- **FreeSolv** — total ?; splits: test (-1)
- **CEP** — total ?; splits: test (-1)
- **BetaLactamase** — total ?; splits: test (-1)
- **Stability** — total ?; splits: test (-1)
- **BindingDB** — total ?; splits: test (-1)
- **PPIAffinity** — total ?; splits: test (-1)
- **BBBP** — total ?; splits: test (-1)
- **GO-CC** — total ?; splits: test (-1)
- **DrugBank** — total ?; splits: test (-1)
- **HumanPPI** — total ?; splits: test (-1)
- **YeastPPI** — total ?; splits: test (-1)

## Metrics

- `Root Mean Square Error (RMSE)` **(primary)** — range: other
  - The square root of the average of squared differences between predicted and actual values. Lower values indicate better performance.
- `Area Under the Curve (AUC)` — range: [0, 1]
  - The area under the Receiver Operating Characteristic (ROC) curve, measuring the model's ability to distinguish between classes across all classification thresholds. Higher values indicate better performance.

## Input / output format

**Input**: Protein sequences/structures and molecular graphs/SMILES paired via biochemical reaction networks.

**Output**: Continuous values for regression tasks or class probabilities/labels for classification tasks.

## Scoring recipe

```python
def rmse(y_true, y_pred):
    return np.sqrt(np.mean((y_true - y_pred) ** 2))

def auc(y_true, y_pred):
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(y_true, y_pred)
```

## Common pitfalls

- Positive Δ% values in the tables indicate improvement (reduction in RMSE for regression), which is counterintuitive.
- Evaluations compare against multiple baseline combinations (ESM3/GearNet/ProtBert × MolCLR/MolFormer) rather than a single fixed baseline, complicating direct cross-task comparisons.
- Statistical significance is reported at p < 0.05, but the specific statistical test used (e.g., paired t-test) is not detailed in the results section.

## Evidence (verbatim from paper)

> Table 1: Evaluation of ReactEmbed against pre-trained baselines for regression tasks, using Root Mean Square Error (RMSE) as the evaluation metric (lower is better). The Δ% values show the absolute change and relative percentage improvement compared to the pre-trained baseline, where positive values indicate improvement (reduction in RMSE).

## Citation

```bibtex
@misc{sicherman2025reactembed,
  title={ReactEmbed: A Cross-Domain Framework for Protein-Molecule Representation Learning via Biochemical Reaction Networks},
  author={Sicherman et al. (2025)},
  year={2025},
  note={arXiv:2501.18278}
}
```

- arXiv: 2501.18278