Madrid Traffic Forecasting Eval

---
name: madrid-traffic-forecasting-eval
description: Evaluates the short-term traffic flow forecasting capability of various machine learning and deep learning models on real-world urban road networks. It specifically probes how well models generalize across different traffic profiles and prediction horizons while maintaining computational efficiency. Use when the user wants to benchmark on Madrid Traffic Dataset, or asks about evaluating this task. Reports R^2 (Coefficient of determination).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.08170
  bibtex_key: delser2020deep
  confidence: high
---

# madrid-traffic-forecasting-eval

> Deep Echo State Networks for Short-Term Traffic Forecasting: Performance Comparison and Statistical Assessment — Del Ser et al. (2020) (arXiv:2004.08170, 2020)

## What this evaluates

Evaluates the short-term traffic flow forecasting capability of various machine learning and deep learning models on real-world urban road networks. It specifically probes how well models generalize across different traffic profiles and prediction horizons while maintaining computational efficiency.

## Datasets

- **Madrid Traffic Dataset** — total 133; splits: 10-fold time-split (-1)

## Metrics

- `R^2 (Coefficient of determination)` **(primary)** — range: [-inf, 1]
  - Calculated as 1 - (sum((y_true - y_pred)^2) / sum((y_true - mean(y_true))^2)), measuring the proportion of variance in traffic flow explained by the model.

## Input / output format

**Input**: Traffic flow time series aggregated every 15 minutes, transformed into supervised learning instances using a sliding window of W=6 past samples.

**Output**: Predicted traffic flow value for a given prediction horizon h ∈ {1, 2, 3, 4} slots.

## Scoring recipe

```python
def compute_r2(y_true, y_pred):
    ss_res = sum((y_true - y_pred) ** 2)
    ss_tot = sum((y_true - mean(y_true)) ** 2)
    return 1 - (ss_res / ss_tot)

# For each ATR and horizon h:
for a in range(133):
    scores = []
    for p in range(10):  # 10 time-split partitions
        train, test = split_time_series(data[a], p)
        model.fit(train)
        y_pred = model.predict(test)
        scores.append(compute_r2(test.targets, y_pred))
    avg_score[a] = mean(scores)

# Statistical ranking across M models:
# 1. Friedman test on avg_scores across models
# 2. If significant, Wilcoxon signed-rank test for pairwise comparisons
# 3. Count WINS, TIES, LOSSES per model
# 4. Compute fractional rankings based on WINS/TIES
# 5. Average ranks across all 133 ATRs
# 6. Compute Nemenyi Critical Distance: CD = Q * sqrt(M*(M+1)/(6*N_ATR))
```

## Common pitfalls

- Using random k-fold cross-validation instead of chronological time-split splits, which leaks future information into training.
- Ignoring the sliding window transformation (W=6) when preparing input sequences, leading to shape mismatches.
- Applying standard accuracy metrics (e.g., MAE, RMSE) without converting to R^2 or failing to report statistical significance via Friedman/Wilcoxon tests.

## Evidence (verbatim from paper)

> Each hyper-parameter combination was evaluated over a separate subset of 10 ATRs in terms of their average coefficient of determination (R^2). ... Given a prediction horizon h and an ATR, R^2 scores were recorded for every model over 10 train/test time-split partitions... two statistical hypothesis tests were applied at a significance level alpha=0.05: 1. A Friedman test, to enforce an initial check whether statistically significant gaps are present... 2. A Wilcoxon signed rank test, to assess the relevance of the difference among means in pairwise comparisons...

## Citation

```bibtex
@misc{delser2020deep,
  title={Deep Echo State Networks for Short-Term Traffic Forecasting: Performance Comparison and Statistical Assessment},
  author={Del Ser et al. (2020)},
  year={2020},
  note={arXiv:2004.08170}
}
```

- arXiv: 2004.08170