Ot Detection Eval

---
name: ot-detection-eval
description: Evaluates a machine learning model's ability to detect overshooting tops (OTs) in satellite imagery at a 2 km pixel resolution. It measures how well the model predicts convection/OT presence using physics-informed features derived from visible and infrared channels. Use when the user wants to benchmark on GOES-16 ABI + MRMS Convection Labels, or asks about evaluating this task. Reports hit, correct rejection, false alarm, miss counts.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.03183
  bibtex_key: mitchell2025transparent
  confidence: high
---

# ot-detection-eval

> Transparent Machine Learning: Training and Refining an Explainable Boosting Machine to Identify Overshooting Tops in Satellite Imagery — Mitchell et al. (2025) (arXiv:2507.03183, 2025)

## What this evaluates

Evaluates a machine learning model's ability to detect overshooting tops (OTs) in satellite imagery at a 2 km pixel resolution. It measures how well the model predicts convection/OT presence using physics-informed features derived from visible and infrared channels.

## Datasets

- **GOES-16 ABI + MRMS Convection Labels** — total ?; splits: test (2619)

## Metrics

- `hit, correct rejection, false alarm, miss counts` **(primary)** — range: count
  - Pixel-level binary classification evaluated via confusion matrix counts: hit (model=1, label=1), correct rejection (model=0, label=0), false alarm (model=1, label=0), miss (model=0, label=1). Accuracy is derived as (hits + correct rejections) / total pixels.

## Input / output format

**Input**: Per-pixel features extracted from GOES-16 ABI visible (Channel 2) and infrared (Channel 13) imagery, including brightness, infrared temperature, and Gray-Level Co-occurrence Matrix texture features.

**Output**: Binary classification per pixel indicating presence (1) or absence (0) of convection/overshooting top.

## Scoring recipe

```python
hits = 0; correct_rejections = 0; false_alarms = 0; misses = 0
for pred, label in zip(predictions, labels):
    if pred == 1 and label == 1: hits += 1
    elif pred == 0 and label == 0: correct_rejections += 1
    elif pred == 1 and label == 0: false_alarms += 1
    elif pred == 0 and label == 1: misses += 1
total = hits + correct_rejections + false_alarms + misses
accuracy = (hits + correct_rejections) / total
```

## Common pitfalls

- Labels are derived from MRMS convection flags, not explicit OT annotations, so metrics actually measure convection prediction rather than OT detection.
- The 2 km prediction grid inherently causes spatial mismatch with labels, meaning some errors are unavoidable and not indicative of poor model performance.
- The VIS/IR sandwich product is used only for qualitative intuition and must not be used for quantitative validation.

## Evidence (verbatim from paper)

> When a prediction is made by the model, it is classified as one of the following: a “hit” if both the model and the convection labels indicated the presence of convection, a “correct rejection” if both the model and the convection labels indicated a lack of convection, a “false alarm” if the model indicated the presence of convection but the convection labels did not, and a “miss” if the model indicated a lack of convection but the convection labels did not. The finalized model achieved 30,755 hits, 10,481,845 correct rejections, 32,424 false alarms, and 182,400 misses.

## Citation

```bibtex
@misc{mitchell2025transparent,
  title={Transparent Machine Learning: Training and Refining an Explainable Boosting Machine to Identify Overshooting Tops in Satellite Imagery},
  author={Mitchell et al. (2025)},
  year={2025},
  note={arXiv:2507.03183}
}
```

- arXiv: 2507.03183