Tox21 Fsl Eval

---
name: tox21-fsl-eval
description: Evaluates graph neural networks for few-shot toxic molecule classification. It probes the model's ability to generalize from very limited labeled examples (shots) and adapt to new query sets using meta-learning and graph augmentation techniques. Use when the user wants to benchmark on Tox21, or asks about evaluating this task. Reports ROC-AUC Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.13490
  bibtex_key: mehta2023benchmarking
  confidence: high
---

# tox21-fsl-eval

> Benchmarking Toxic Molecule Classification using Graph Neural Networks and Few Shot Learning — Mehta et al. (2023) (arXiv:2311.13490, 2023)

## What this evaluates

Evaluates graph neural networks for few-shot toxic molecule classification. It probes the model's ability to generalize from very limited labeled examples (shots) and adapt to new query sets using meta-learning and graph augmentation techniques.

## Datasets

- **Tox21** — total ?; splits: train (-1), val (-1)

## Metrics

- `ROC-AUC Score` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `Validation Accuracy` — range: percent
  - Percentage of correctly classified instances in the validation set.

## Input / output format

**Input**: Molecular graphs represented as node and edge feature matrices, processed under a few-shot learning protocol with 10 training shots and 15 query examples per task.

**Output**: Predicted toxicity class label (or probability scores) for each molecule in the query set.

## Scoring recipe

```python
def compute_metrics(preds, labels):
    acc = np.mean(preds == labels) * 100
    fpr, tpr, _ = roc_curve(labels, preds[:, 1])
    auc = auc(fpr, tpr)
    return acc, auc
```

## Common pitfalls

- Few-shot splits are highly sensitive to the specific selection of the 10-shot training set and 15-query examples; results may vary significantly across different random seeds.
- GNN performance is heavily influenced by node ordering and graph isomorphism; models like GCN are sensitive to permutation while GIN is not, affecting reproducibility if graph canonicalization is skipped.
- Class imbalance in toxic molecule datasets can skew accuracy; ROC-AUC is a more robust metric but requires proper probability calibration.

## Evidence (verbatim from paper)

> The baseline model, GCN, achieves a validation accuracy of 65.02% and a ROC-AUC score of 0.732. Comparatively, the GCN+FLAG model showcases an improvement with a validation accuracy of 70.68% (+5.66%) and a ROC-AUC score of 0.806 (+0.074). The GIN model demonstrates the highest validation accuracy of 73.23% (+8.21%) and a ROC-AUC score of 0.816 (+0.084).

## Citation

```bibtex
@misc{mehta2023benchmarking,
  title={Benchmarking Toxic Molecule Classification using Graph Neural Networks and Few Shot Learning},
  author={Mehta et al. (2023)},
  year={2023},
  note={arXiv:2311.13490}
}
```

- arXiv: 2311.13490