Stgformer Eval

---
name: stgformer-eval
description: This evaluation protocol tests a model's ability to forecast future traffic flow across large-scale urban road networks using historical spatiotemporal sensor data. It probes the model's capacity to capture complex spatial dependencies and temporal dynamics while maintaining computational efficiency on real-world traffic benchmarks. Use when the user wants to benchmark on LargeST, PEMS-series, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.00385
  bibtex_key: wang2024stgformer
  confidence: high
---

# stgformer-eval

> STGformer: Efficient Spatiotemporal Graph Transformer for Traffic Forecasting — Wang et al. (2024) (arXiv:2410.00385, 2024)

## What this evaluates

This evaluation protocol tests a model's ability to forecast future traffic flow across large-scale urban road networks using historical spatiotemporal sensor data. It probes the model's capacity to capture complex spatial dependencies and temporal dynamics while maintaining computational efficiency on real-world traffic benchmarks.

## Datasets

- **LargeST** — total ?; splits: train (-1), val (-1), test (-1)
- **PEMS-series** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `MAE` **(primary)** — range: [0, ∞)
  - Mean Absolute Error: average of absolute differences between predicted and true traffic flow values. Computed only on non-zero entries to exclude noisy data.
- `RMSE` — range: [0, ∞)
  - Mask-Based Root Mean Square Error: square root of the average squared differences between predicted and true values, excluding zero entries.
- `MAPE` — range: [0, 100] percent
  - Mean Absolute Percentage Error: average of absolute percentage errors between predicted and true values, excluding zero entries.

## Input / output format

**Input**: Historical traffic flow sequences (12-step windows at 15-minute intervals for LargeST; 5-minute intervals for PEMS-series) representing sensor readings across California regions.

**Output**: Predicted traffic flow values for future time steps (horizons 3, 6, and 12).

## Scoring recipe

```python
def compute_metrics(pred, true):
    mask = true != 0
    pred_m, true_m = pred[mask], true[mask]
    mae = np.mean(np.abs(pred_m - true_m))
    rmse = np.sqrt(np.mean((pred_m - true_m) ** 2))
    mape = np.mean(np.abs((true_m - pred_m) / true_m)) * 100
    return mae, rmse, mape
```

## Common pitfalls

- Forgetting to mask out zero values (indicating noisy/missing data) before computing metrics, which drastically inflates MAPE and RMSE.
- Using random data splits instead of the specified chronological 6:2:2 ratio, violating the temporal nature of traffic forecasting.
- Mismatching horizon step definitions across datasets (15-min intervals for LargeST vs 5-min for PEMS-series).

## Evidence (verbatim from paper)

> In our experiments, we assess model performance using the Mask-Based Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE) as metrics, wherein zero values (indicating noisy data) are disregarded. Data were partitioned chronologically into training, validation, and test sets at a 6:2:2 ratio across all sub-datasets.

## Citation

```bibtex
@misc{wang2024stgformer,
  title={STGformer: Efficient Spatiotemporal Graph Transformer for Traffic Forecasting},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2410.00385}
}
```

- arXiv: 2410.00385