Uncertainty

---
name: uncertainty
description: Evaluates a CNN's ability to predict stellar atmospheric parameters and chemical abundances from low-resolution spectra, measuring both internal consistency across model runs and agreement with established spectroscopic pipeline measurements. Use when the user has predictions and gold and needs to compute Uncertainty.
metadata:
  skill_kind: metric
  source_arxiv: 2208.08872
  bibtex_key: [REDACTED]
  confidence: high
---

# uncertainty

> The Gaia-ESO Survey: Chemical evolution of Mg and Al in the Milky Way with Machine-Learning — Ambrosch et al. (2022) (arXiv:2208.08872, 2022)

## What this evaluates

Evaluates a CNN's ability to predict stellar atmospheric parameters and chemical abundances from low-resolution spectra, measuring both internal consistency across model runs and agreement with established spectroscopic pipeline measurements.

## Datasets

- **Gaia-ESO Survey (GES)** — total 30000; splits: train (-1), test (-1), observed (-1)

## Metrics

- `Uncertainty` **(primary)** — range: K or dex
  - Standard deviation of predictions across eight independently trained CNN models with different random weight initializations.
- `Bias & Scatter` — range: K or dex
  - Mean difference and standard deviation around the 1:1 relation between CNN predictions and Gaia-ESO pipeline measurements.

## Input / output format

**Input**: GIRAFFE low-resolution stellar spectra (flux as a function of wavelength).

**Output**: Five continuous values: T_eff (K), log g (dex), [Mg/Fe] (dex), [Al/Fe] (dex), [Fe/H] (dex).

## Scoring recipe

```python
# preds: list of 8 model predictions per star
# gold: GES pipeline measurements
internal_uncertainty = np.std(preds, axis=0)
bias = np.mean(preds[0] - gold)
scatter = np.std(preds[0] - gold)
```

## Common pitfalls

- Internal uncertainty measures model initialization variance, not data noise or true prediction error.
- Performance degrades significantly for low S/N spectra (<30) and extreme abundance values due to limited training data.
- Strong label correlations in training data can cause the network to learn spurious correlations instead of physical spectral features.

## Evidence (verbatim from paper)

> We define the uncertainties of our results as the dispersion between the label predictions from the eight CNN models. In Fig. 8 we display the distribution of the label uncertainties σ(Label) relative to the predicted label values of our five labels. ... The mean uncertainties of the label predictions are small: 24K for T_eff, 0.03 for log(g), 0.02 dex for [Mg/Fe], 0.03 dex for [Al/Fe], and 0.02 dex for [Fe/H].

## Citation

```bibtex
@misc{ambrosch2022gaia,
  title={The Gaia-ESO Survey: Chemical evolution of Mg and Al in the Milky Way with Machine-Learning},
  author={Ambrosch et al. (2022)},
  year={2022},
  note={arXiv:2208.08872}
}
```

- arXiv: 2208.08872