Droughted Eval

---
name: droughted-eval
description: Evaluates time-series forecasting models on predicting U.S. drought severity across 1 to 6 week horizons using meteorological and static features. It probes both regression accuracy and multi-class classification performance for drought monitoring levels. Use when the user wants to benchmark on DroughtED, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.07012
  bibtex_key: duan2022automl
  confidence: high
---

# droughted-eval

> AutoML-Based Drought Forecast with Meteorological Variables — Duan et al. (2022) (arXiv:2207.07012, 2022)

## What this evaluates

Evaluates time-series forecasting models on predicting U.S. drought severity across 1 to 6 week horizons using meteorological and static features. It probes both regression accuracy and multi-class classification performance for drought monitoring levels.

## Datasets

- **DroughtED** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error: the average of the absolute differences between predicted and actual drought monitor values.
- `macro-averaged F1` — range: [0, 1]
  - Macro-averaged F1 score computed over classes 0–5. For regression outputs, predictions are rounded to the nearest integer before computing F1 against rounded gold labels.

## Input / output format

**Input**: Normalized meteorological variables (scaled by median and interquartile range) and static features (scaled by mean and standard deviation), along with target drought monitor values (0–5).

**Output**: Regression: continuous float value. Classification: discrete integer class (0–5). F1 can also be computed from regression outputs by rounding both predictions and gold labels.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    mae = np.mean(np.abs(preds - golds))
    rounded_preds = np.round(preds)
    rounded_golds = np.round(golds)
    f1 = f1_score(rounded_golds, rounded_preds, average='macro')
    return mae, f1
```

## Common pitfalls

- The authors deviate from the original benchmark's validation split (using 60/40 train/val instead of a fixed 1-year validation set), which the authors themselves note makes direct comparison unfair.
- The benchmark originally assessed classification via post-processing of regression outputs, whereas this evaluation trains separate classification models, creating a methodological mismatch.
- F1 scores for regression models are derived by rounding continuous predictions and gold labels, which can artificially inflate or deflate performance compared to direct classification training.

## Evidence (verbatim from paper)

> The DroughtED benchmark compares F1 score and mean absolute errors (MAE) for Week1 to Week6 forecasts. In this paper, the model is trained in a different manner with the benchmark. Two models are trained: regression (for MAE comparisons) and classification (for F1 comparisons), while the benchmark DL models are trained with a regression loss and the classification performance is assessed with post-processing. In regression, the evaluation metrics is set to MAE, and macro-averaged F1 for classification.

## Citation

```bibtex
@misc{duan2022automl,
  title={AutoML-Based Drought Forecast with Meteorological Variables},
  author={Duan et al. (2022)},
  year={2022},
  note={arXiv:2207.07012}
}
```

- arXiv: 2207.07012