Integrated Brier Score

---
name: integrated-brier-score
description: Evaluates the calibration and predictive accuracy of ensemble forecasting methods for time-to-event outcomes in meteorology. It compares how well different combination techniques predict the timing of events like the first hard freeze. Use when the user has predictions and gold and needs to compute Mean Integrated Brier Score (IBS).
metadata:
  skill_kind: metric
  source_arxiv: 2503.19534
  bibtex_key: [REDACTED]
  confidence: medium
---

# integrated-brier-score

> Combining predictive distributions for time-to-event outcomes in meteorology — Cunen et al. (2025) (arXiv:2503.19534, 2025)

## What this evaluates

Evaluates the calibration and predictive accuracy of ensemble forecasting methods for time-to-event outcomes in meteorology. It compares how well different combination techniques predict the timing of events like the first hard freeze.

## Datasets

- **Meteorological time-to-event scenarios** — total ?; splits: test (-1)

## Metrics

- `Mean PIT` — range: [0, 1]
  - Probability Integral Transform values averaged across scenarios. For well-calibrated forecasts, PIT values should follow a uniform distribution on [0,1], so the mean should be close to 0.5.
- `Standard deviation of PIT` — range: [0, 1]
  - Standard deviation of PIT values across scenarios. Values closer to the theoretical uniform standard deviation (~0.289) indicate better calibration.
- `Mean Integrated Brier Score (IBS)` **(primary)** — range: [0, 1]
  - Integrated Brier Score averaged across scenarios. Measures the accuracy of predictive survival distributions; lower values indicate better predictive performance.

## Input / output format

**Input**: Predictive survival distributions (or ensemble forecasts) for time-to-event outcomes, along with observed event times.

**Output**: Point estimates or full predictive distributions for the time until the first hard freeze (or similar time-to-event outcome).

## Scoring recipe

```python
For each scenario:
  pit_values = [F_i(t_i) for i in instances]  # F_i is predictive CDF, t_i is observed time
  mean_pit = mean(pit_values)
  std_pit = std(pit_values)
  ibs = integrated_brier_score(predictive_distributions, observed_times)
Return mean_pit, std_pit, and mean(ibs) across scenarios.
```

## Common pitfalls

- PIT values near 0.5 indicate good calibration, but deviations can be subtle and require many scenarios to detect reliably.
- IBS is a proper scoring rule where lower is better; readers may mistakenly interpret higher scores as better performance.
- The evaluation aggregates results across 16 heterogeneous scenarios, which may mask method performance on specific meteorological conditions.

## Evidence (verbatim from paper)

> Table A3: Mean Integrated Brier scores (IBS) for 17 forecasting methods in 16 different scenarios. The lowest scores in each scenario are bold-faced.

## Citation

```bibtex
@misc{cunen2025combining,
  title={Combining predictive distributions for time-to-event outcomes in meteorology},
  author={Cunen et al. (2025)},
  year={2025},
  note={arXiv:2503.19534}
}
```

- arXiv: 2503.19534