Tox21 Challenge Eval

---
name: tox21-challenge-eval
description: Evaluates molecular toxicity prediction capabilities across diverse AI architectures (descriptor-based models, neural networks, tabular transformers, and zero-shot LLMs) on a standardized chemical safety benchmark. Use when the user wants to benchmark on Tox21 Challenge dataset, or asks about evaluating this task. Reports performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.14744
  bibtex_key: ebner2025tox21leaderboard
  confidence: medium
---

# tox21-challenge-eval

> Measuring AI Progress in Drug Discovery: A Reproducible Leaderboard for the Tox21 Challenge — Ebner et al. (2025) (arXiv:2511.14744, 2025)

## What this evaluates

Evaluates molecular toxicity prediction capabilities across diverse AI architectures (descriptor-based models, neural networks, tabular transformers, and zero-shot LLMs) on a standardized chemical safety benchmark.

## Datasets

- **Tox21 Challenge dataset** — total ?; splits: test (-1)

## Metrics

- `performance` **(primary)** — range: [0, 1]
  - Unspecified in the provided text; typically corresponds to the area under the receiver operating characteristic curve (AUC-ROC) averaged across the 12 Tox21 targets in standard benchmarks.

## Input / output format

**Input**: Molecular structures represented as SMILES strings or tabular molecular descriptors.

**Output**: Predicted toxicity scores or binary classification labels for each target.

## Scoring recipe

```python
def compute_metric(predictions, gold):
    # Metric name unspecified in text; standard Tox21 uses AUC-ROC per target
    scores = []
    for target in targets:
        scores.append(auroc(gold[target], predictions[target]))
    return mean(scores)
# Note: Paper reports median across 5 runs for local evaluation, single run for leaderboard.
```

## Common pitfalls

- Using altered or downstream versions of the dataset (e.g., MoleculeNet, OGB) instead of the original, unaltered Tox21 Challenge dataset.
- Assuming leaderboard scores are deterministic; the paper explicitly notes hardware- and system-level nondeterminism causes discrepancies between local median runs and single-run leaderboard scores.

## Evidence (verbatim from paper)

> Table[1] summarizes the performance of our baseline toxicity prediction models on the original Tox21-Challenge test set. The results in Table[1] differ slightly from those on the Hugging Face leaderboard. Table[1] reports the median performance across five independent runs conducted on our local infrastructure, whereas the leaderboard shows the score from a single run of the median-performing model, evaluated via the standardized Hugging Face pipeline.

## Citation

```bibtex
@misc{ebner2025tox21leaderboard,
  title={Measuring AI Progress in Drug Discovery: A Reproducible Leaderboard for the Tox21 Challenge},
  author={Ebner et al. (2025)},
  year={2025},
  note={arXiv:2511.14744}
}
```

- arXiv: 2511.14744