Gess Ood Eval

---
name: gess-ood-eval
description: Evaluates geometric deep learning models' out-of-distribution generalization across scientific domains under conditional, covariate, and concept shifts. It probes how different learning paradigms (ERM, domain adaptation, transfer learning, and OOD generalization) perform when provided with varying amounts of target-domain data. Use when the user wants to benchmark on Track (Particle Tracking Simulation), QMOF (Quantum Metal-organic Frameworks), DrugOOD-3D (3D Conformers of Drug Molecules), or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.08677
  bibtex_key: zou2023gess
  confidence: medium
---

# gess-ood-eval

> GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts — Zou et al. (2023) (arXiv:2310.08677, 2023)

## What this evaluates

Evaluates geometric deep learning models' out-of-distribution generalization across scientific domains under conditional, covariate, and concept shifts. It probes how different learning paradigms (ERM, domain adaptation, transfer learning, and OOD generalization) perform when provided with varying amounts of target-domain data.

## Datasets

- **Track (Particle Tracking Simulation)** — total ?; splits: Train-ID (-1), Val-ID (-1), Test-ID (-1), Train-OOD (-1), Val-OOD (-1), Test-OOD (-1)
- **QMOF (Quantum Metal-organic Frameworks)** — total ?; splits: Train-ID (-1), Val-ID (-1), Test-ID (-1), Train-OOD (-1), Val-OOD (-1), Test-OOD (-1)
- **DrugOOD-3D (3D Conformers of Drug Molecules)** — total ?; splits: Train-ID (-1), Val-ID (-1), Test-ID (-1), Train-OOD (-1), Val-OOD (-1), Test-OOD (-1)

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error, calculated as the average of the absolute differences between predicted and true values across all test instances.

## Input / output format

**Input**: Geometric or graph-structured data representing scientific objects (e.g., particle collision events, molecular frameworks, drug conformers) with associated node/edge features and target labels.

**Output**: Predicted continuous values (regression) or class labels corresponding to the input geometric structures.

## Scoring recipe

```python
def compute_mae(predictions, gold_labels):
    errors = [abs(p - g) for p, g in zip(predictions, gold_labels)]
    return sum(errors) / len(errors)
```

## Common pitfalls

- Assuming OOD generalization methods (trained only on ID data) will consistently outperform ERM; the paper shows they often fail to provide significant improvements in practical scientific scenarios.
- Assuming fine-tuning with few OOD labels always improves performance; it can cause catastrophic forgetting or underperform when the distribution shift is mild or labeled data is too scarce.
- Assuming subgroup splits for methods like GroupDRO accurately reflect the true distribution shift; subjective splits may misalign with the actual shift mechanism, degrading robustness.

## Evidence (verbatim from paper)

> We find this strategy performs particularly well in the case of HSE06 (even fine-tuned on a very small number of OOD labels), where it largely outperforms all other methods with the MAE score increased by at least 30%.

## Citation

```bibtex
@misc{zou2023gess,
  title={GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts},
  author={Zou et al. (2023)},
  year={2023},
  note={arXiv:2310.08677}
}
```

- arXiv: 2310.08677