Molecular Property Benchmark Eval

---
name: molecular-property-benchmark-eval
description: Evaluates the predictive performance of various machine learning architectures (deep vs. non-deep) on molecular property prediction tasks. It probes how well models handle irregular, tree-like molecular data patterns across both classification and regression benchmarks. Use when the user wants to benchmark on BACE, HIV, BBBP, ClinTox, SIDER, Tox21, ToxCast, MUV, SARS-CoV-2, ESOL, Lipop, FreeSolv, QM7, QM8, or asks about evaluating this task. Reports AUC_ROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.17702
  bibtex_key: xia2023deepmodels
  confidence: high
---

# molecular-property-benchmark-eval

> Why Deep Models Often cannot Beat Non-deep Counterparts on Molecular Property Prediction? — Xia et al. (2023) (arXiv:2306.17702, 2023)

## What this evaluates

Evaluates the predictive performance of various machine learning architectures (deep vs. non-deep) on molecular property prediction tasks. It probes how well models handle irregular, tree-like molecular data patterns across both classification and regression benchmarks.

## Datasets

- **BACE** — total 1513; splits: test (-1)
- **HIV** — total 40748; splits: test (-1)
- **BBBP** — total 2035; splits: test (-1)
- **ClinTox** — total 1475; splits: test (-1)
- **SIDER** — total 1366; splits: test (-1)
- **Tox21** — total 7811; splits: test (-1)
- **ToxCast** — total 8539; splits: test (-1)
- **MUV** — total 93087; splits: test (-1)
- **SARS-CoV-2** — total 14332; splits: test (-1)
- **ESOL** — total 1127; splits: test (-1)
- **Lipop** — total 4200; splits: test (-1)
- **FreeSolv** — total 639; splits: test (-1)
- **QM7** — total 6830; splits: test (-1)
- **QM8** — total 21786; splits: test (-1)

## Metrics

- `AUC_ROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds.
- `AUC_PRC` — range: [0, 1]
  - Area under the Precision-Recall Curve. Evaluates model performance on imbalanced classification tasks by plotting precision against recall.
- `RMSE` — range: other
  - Root Mean Squared Error. Calculated as the square root of the average of squared differences between predicted and actual continuous values.
- `MAE` — range: other
  - Mean Absolute Error. Calculated as the average of absolute differences between predicted and actual continuous values.

## Input / output format

**Input**: Molecular structures provided as SMILES strings or graph representations, converted into fixed-length feature vectors or fingerprints for model ingestion.

**Output**: Continuous values for regression tasks (e.g., solubility, energy) or probability scores/labels for classification tasks (e.g., drug toxicity, blood-brain barrier penetration).

## Scoring recipe

```python
def evaluate(y_true, y_pred, task_type):
    if task_type == 'classification':
        auc_roc = roc_auc_score(y_true, y_pred)
        auc_prc = average_precision_score(y_true, y_pred)
        return {'AUC_ROC': auc_roc, 'AUC_PRC': auc_prc}
    elif task_type == 'regression':
        rmse = np.sqrt(mean_squared_error(y_true, y_pred))
        mae = mean_absolute_error(y_true, y_pred)
        return {'RMSE': rmse, 'MAE': mae}
```

## Common pitfalls

- Assuming dataset size is the primary bottleneck for deep learning, whereas the paper demonstrates that irregular/tree-like data patterns are the dominant factor.
- Comparing raw metric values across datasets without accounting for the fact that different datasets use different evaluation metrics (AUC_ROC, AUC_PRC, RMSE, MAE).
- Overlooking that the pre-trained transformer (TRSF) leverages 861,000 pre-training SMILES strings, giving it a data advantage not available to the non-deep baselines.

## Evidence (verbatim from paper)

> Table 1: The comparison of representative models on multiple molecular datasets. ... BACE (1,513) | AUC_ROC | 0.886 | 0.896 | 0.890 | 0.815 | 0.559 | 0.835 | 0.887 | 0.880 | 0.846 | 0.886 | 0.879 | 0.882 |

## Citation

```bibtex
@misc{xia2023deepmodels,
  title={Why Deep Models Often cannot Beat Non-deep Counterparts on Molecular Property Prediction?},
  author={Xia et al. (2023)},
  year={2023},
  note={arXiv:2306.17702}
}
```

- arXiv: 2306.17702