Methane Segmentation Eval

---
name: methane-segmentation-eval
description: Evaluates the capability of hyperspectral image processing models to detect and segment methane plumes on resource-constrained satellite hardware. It probes the trade-off between detection accuracy (precision, recall, F1) and computational efficiency (runtime) across different spectral enhancement filters and lightweight neural networks. Use when the user wants to benchmark on STARCOP, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.01472
  bibtex_key: herec2025methane
  confidence: high
---

# methane-segmentation-eval

> Optimizing Methane Detection On Board Satellites: Speed, Accuracy, and Low-Power Solutions for Resource-Constrained Hardware — Herec et al. (2025) (arXiv:2507.01472, 2025)

## What this evaluates

Evaluates the capability of hyperspectral image processing models to detect and segment methane plumes on resource-constrained satellite hardware. It probes the trade-off between detection accuracy (precision, recall, F1) and computational efficiency (runtime) across different spectral enhancement filters and lightweight neural networks.

## Datasets

- **STARCOP** — total ?; splits: test (-1); repo https://github.com/zaitra/methane-filters-benchmark

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Precision is TP/(TP+FP), Recall is TP/(TP+FN).
- `F1 - Strong` — range: [0, 1]
  - F1 score computed exclusively on instances or tiles annotated as strong methane plumes.

## Input / output format

**Input**: 512x512 pixel tiles of hyperspectral imagery (72 channels) concatenated with RGB bands.

**Output**: Binary segmentation mask indicating methane plume presence per pixel.

## Scoring recipe

```python
def compute_f1(pred_mask, gold_mask):
    tp = np.sum((pred_mask == 1) & (gold_mask == 1))
    fp = np.sum((pred_mask == 1) & (gold_mask == 0))
    fn = np.sum((pred_mask == 0) & (gold_mask == 1))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
# For F1 - Strong, apply compute_f1 only to regions annotated as strong plumes.
```

## Common pitfalls

- Original Mag1c runtime is measured on 512x512 tiles but the STARCOP dataset provides precomputed column-wise scene products, making its reported runtime an aspirational upper bound rather than a strict baseline.
- Metrics are averaged over 5 repeated training runs with standard deviations reported; ignoring variance may misrepresent model stability.
- F1 - Strong isolates performance on strong plumes only, potentially masking poor detection on weak or medium plumes.

## Evidence (verbatim from paper)

> The original Mag1c method achieves the highest F1 score for strong plumes (67.50 %) but has a very long runtime (109.61 s). TABLE I: Results showing runtimes and metrics for 512x512 tiles with 72 channels. The total runtime is determined by adding the inference time to the optimized runtime (or, if an optimization was not implemented, the original runtime). For entries that do not mention an ML model, the morphological baseline was applied for inference. The average score and standard deviation are shown for 5 repeated training runs.

## Citation

```bibtex
@misc{herec2025methane,
  title={Optimizing Methane Detection On Board Satellites: Speed, Accuracy, and Low-Power Solutions for Resource-Constrained Hardware},
  author={Herec et al. (2025)},
  year={2025},
  note={arXiv:2507.01472}
}
```

- arXiv: 2507.01472