Qm9 Mol Structtok Eval

---
name: qm9-mol-structtok-eval
description: Evaluates the ability of a tokenization framework to generate valid 3D molecular structures and predict quantum mechanical properties. It probes structural validity, geometric plausibility, conditional controllability, and property prediction accuracy on organic molecules. Use when the user wants to benchmark on QM9, or asks about evaluating this task. Reports Mean Absolute Error (MAE).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.01564
  bibtex_key: gao2024tokenizing
  confidence: high
---

# qm9-mol-structtok-eval

> Tokenizing 3D Molecule Structure with Quantized Spherical Coordinates — Gao et al. (2024) (arXiv:2412.01564, 2024)

## What this evaluates

Evaluates the ability of a tokenization framework to generate valid 3D molecular structures and predict quantum mechanical properties. It probes structural validity, geometric plausibility, conditional controllability, and property prediction accuracy on organic molecules.

## Datasets

- **QM9** — total 134000; splits: train (110000), val (10000), test (10831)

## Metrics

- `Mean Absolute Error (MAE)` **(primary)** — range: other
  - MAE = (1/N) * Σ|y_true - y_pred|, where y_true is the target quantum property and y_pred is the predicted property of the generated molecule. Lower values indicate better controllability and prediction accuracy.
- `Validity Rate` — range: percent
  - Percentage of generated molecules that successfully pass bond assignment via xyz2mol or OpenBabel without charge imbalances, or convert to valid SMILES strings.
- `Uniqueness Rate` — range: percent
  - Proportion of generated molecules that have unique SMILES representations, calculated as 1 - (number of duplicate structures / total generated molecules).
- `PoseBusters Geometric Checks` — range: percent
  - Pass rate (%) across seven geometric and energetic sanity checks: All Atoms Connected, Reasonable Bond Angles, Reasonable Bond Lengths, Aromatic Ring Flatness, Double Bond Flatness, Reasonable Internal Energy, and No Internal Steric Clash.

## Input / output format

**Input**: 3D atomic coordinates and atom types; SELFIES string representations for autoregressive generation; scalar property values (e.g., polarizability, HOMO/LUMO energies) for conditional generation.

**Output**: Discrete structural tokens (VQ-VAE latent codes), 3D molecular coordinates (xyz), or predicted scalar property values.

## Scoring recipe

```python
def compute_metrics(generated_mols, y_true, y_pred):
    # Validity
    valid_count = sum(1 for m in generated_mols if xyz2mol_or_openbabel(m) and not has_charge_imbalance(m))
    validity_rate = valid_count / len(generated_mols)
    # Uniqueness
    smiles_list = [mol_to_smiles(m) for m in generated_mols]
    uniqueness_rate = 1 - (len(smiles_list) - len(set(smiles_list))) / len(smiles_list)
    # PoseBusters
    posebusters_pass = all([
        check_atoms_connected(m), check_bond_angles(m), check_bond_lengths(m),
        check_aromatic_flatness(m), check_double_bond_flatness(m),
        check_internal_energy(m), check_steric_clash(m)
    ] for m in generated_mols)
    posebusters_rate = sum(posebusters_pass) / len(generated_mols)
    # MAE
    mae = mean(abs(y_true - y_pred))
    return validity_rate, uniqueness_rate, posebusters_rate, mae
```

## Common pitfalls

- Validity is defined by bond assignment tools (xyz2mol/OpenBabel) rather than strict chemical rules, so results may vary depending on the tool version or lookup table used.
- Conditional generation MAE is evaluated on generated molecules using a fixed classifier, not on ground-truth molecules, which can inflate or deflate scores compared to standard property prediction benchmarks.
- PoseBusters was originally designed for protein-ligand complexes; applying its small-molecule-specific metrics may yield conservative or slightly mismatched geometric thresholds.

## Evidence (verbatim from paper)

> The primary evaluation metric was the Mean Absolute Error (MAE) between the given property ξ and the predicted property ξ̂ of the generated molecule. A lower MAE reflects the model's ability to accurately generate molecules with the desired quantum properties.

## Citation

```bibtex
@misc{gao2024tokenizing,
  title={Tokenizing 3D Molecule Structure with Quantized Spherical Coordinates},
  author={Gao et al. (2024)},
  year={2024},
  note={arXiv:2412.01564}
}
```

- arXiv: 2412.01564