Canadafiresat Eval

---
name: canadafiresat-eval
description: Evaluates deep learning models for high-resolution (100 m) wildfire forecasting using multi-modal satellite and environmental data. It probes the model's ability to predict fire occurrence at the patch level across different temporal splits and land cover types, particularly under severe class imbalance and varying fire danger conditions. Use when the user wants to benchmark on CanadaFireSat, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.08690
  bibtex_key: porta2025canadafiresat
  confidence: high
---

# canadafiresat-eval

> CanadaFireSat: Toward high-resolution wildfire forecasting with multiple modalities — Porta et al. (2025) (arXiv:2506.08690, 2025)

## What this evaluates

Evaluates deep learning models for high-resolution (100 m) wildfire forecasting using multi-modal satellite and environmental data. It probes the model's ability to predict fire occurrence at the patch level across different temporal splits and land cover types, particularly under severe class imbalance and varying fire danger conditions.

## Datasets

- **CanadaFireSat** — total ?; splits: train (-1), val (-1), test (-1), test_hard (-1); repo https://github.com/eceo-epfl/CanadaFireSat-Data

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of Precision and Recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Computed at a fixed threshold tuned on the validation set.
- `PRAUC` — range: [0, 1]
  - Area Under the Precision-Recall Curve for the positive class (fire). Summarizes the Precision-Recall trade-off across all thresholds.

## Input / output format

**Input**: Multi-modal temporal inputs per 100m×100m patch: Sentinel-2 satellite image time series and environmental predictors (MODIS/ERA5).

**Output**: Binary classification per patch (fire vs. no fire), binarized at a threshold tuned on the validation set.

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import average_precision_score

def compute_metrics(gold, preds):
    tp = np.sum((preds == 1) & (gold == 1))
    fp = np.sum((preds == 1) & (gold == 0))
    fn = np.sum((preds == 0) & (gold == 1))
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp + fn + 1e-8)
    f1 = 2 * precision * recall / (precision + recall + 1e-8)
    prauc = average_precision_score(gold, preds)
    return f1, prauc
```

## Common pitfalls

- Using accuracy instead of F1/PRAUC due to severe class imbalance between fire and non-fire patches.
- Ignoring the 'Test Hard' split, which uses a different sampling strategy to assess lower-bound performance and models' ability to capture hidden ignition phenomena.
- Overestimating model utility at high FWI values (>20) where the task becomes trivial due to data imbalance, making naive predictors perform well.

## Evidence (verbatim from paper)

> CanadaFireSat covers the period 2016-2023: we train our models on the years 2016-2021, while keeping 2022 for validation (Val) and 2023 for both test sets: Test and Test Hard. Results are evaluated in terms of F1 score and PRAUC (Area Under the Precision-Recall Curve for the positive class fire only). Both metrics are robust to imbalanced datasets, contrarily to patch-level accuracy. The F1 score is defined as the harmonic mean between Precision (proportion of true positive pixels over pixels predicted as positive) and Recall (proportion of true positives over all actual positives): F1 = 2 * (Precision * Recall) / (Precision + Recall).

## Citation

```bibtex
@misc{porta2025canadafiresat,
  title={CanadaFireSat: Toward high-resolution wildfire forecasting with multiple modalities},
  author={Porta et al. (2025)},
  year={2025},
  note={arXiv:2506.08690}
}
```

- arXiv: 2506.08690