Supply Chain Forecasting Eval

---
name: supply-chain-forecasting-eval
description: Evaluates models' ability to generate calibrated probabilistic forecasts of supply chain disruptions from raw news text. It probes temporal generalization, uncertainty quantification, and the prioritization of high-risk signals for decision-making. Use when the user wants to benchmark on Supply Chain Disruption Forecasting Dataset, or asks about evaluating this task. Reports Brier score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.01298
  bibtex_key: [REDACTED]
  confidence: high
---

# supply-chain-forecasting-eval

> Forecasting Supply Chain Disruptions with Foresight Learning — Turtel et al. (2026) (arXiv:2604.01298, 2026)

## What this evaluates

Evaluates models' ability to generate calibrated probabilistic forecasts of supply chain disruptions from raw news text. It probes temporal generalization, uncertainty quantification, and the prioritization of high-risk signals for decision-making.

## Datasets

- **Supply Chain Disruption Forecasting Dataset** — total ?; splits: train (-1), test (-1)

## Metrics

- `Brier score` **(primary)** — range: [0, 1]
  - Mean squared difference between predicted probability and actual binary outcome. Lower values indicate better probabilistic accuracy.
- `Brier skill score (BSS)` — range: percent
  - Percentage improvement in Brier score relative to the historical baseline (training average disruption rate). Positive values indicate better performance than the baseline.
- `Expected Calibration Error (ECE)` — range: [0, 1]
  - Weighted average absolute difference between predicted probabilities and empirical event frequencies across probability bins. Lower values indicate better calibration.
- `Precision@10%` — range: [0, 1]
  - Fraction of true disruption events among the top 10% of predictions ranked by highest predicted probability.

## Input / output format

**Input**: Time-stamped news context and a forecasting question, processed through a fixed prompt template.

**Output**: A single probability value (float) representing the model's predicted likelihood of a supply chain disruption event.

## Scoring recipe

```python
def compute_metrics(predictions, gold, gold_train):
    brier = mean((p - g)**2 for p, g in zip(predictions, gold))
    baseline_rate = mean(gold_train)
    bss = ((baseline_rate * (1 - baseline_rate) - brier) / (baseline_rate * (1 - baseline_rate))) * 100
    bins = 10
    ece = 0.0
    for i in range(bins):
        mask = (predictions >= i/10) & (predictions < (i+1)/10)
        if mask.sum() > 0:
            ece += mask.sum() * abs(predictions[mask].mean() - gold[mask].mean())
    ece /= len(predictions)
    top_k = int(len(predictions) * 0.1)
    top_indices = argsort(predictions, descending=True)[:top_k]
    precision_10 = gold[top_indices].mean()
    return brier, bss, ece, precision_10
```

## Common pitfalls

- Temporal split requirement: The test set is strictly temporally subsequent to training data. Random shuffling causes information leakage and invalidates results.
- BSS baseline dependency: BSS is normalized against the historical baseline (training average rate), not a uniform 0.5 or random guess baseline.
- Precision@10% ranking: Computed on the top 10% of instances by predicted probability, not a fixed top-k count or threshold-based precision.

## Evidence (verbatim from paper)

> Performance is measured using standard probabilistic metrics: Brier score, Brier skill score (BSS), expected calibration error (ECE), and Precision@10%. The Brier score measures the accuracy of probabilistic predictions, while BSS, as measured relative to the historical baseline, captures the percentage improvement in Brier score, with positive values indicating better performance. Precision@10% is defined as the precision among the top 10% of predictions ranked by predicted probability, i.e., the fraction of highest-confidence predictions that correspond to true disruption events.

## Citation

```bibtex
@misc{turtel2026foresight,
  title={Forecasting Supply Chain Disruptions with Foresight Learning},
  author={Turtel et al. (2026)},
  year={2026},
  note={arXiv:2604.01298}
}
```

- arXiv: 2604.01298