Dl Traff Eval

---
name: dl-traff-eval
description: Evaluates the predictive accuracy and computational efficiency of deep learning models for urban traffic forecasting. It benchmarks grid-based, graph-based, and multivariate time-series architectures on standard traffic datasets to compare their ability to capture spatiotemporal dependencies. Use when the user wants to benchmark on BikeNYC-I, TaxiNYC, TaxiBJ, METR-LA, PeMS-BAY, PEMSD7M, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.09091
  bibtex_key: [REDACTED]
  confidence: high
---

# dl-traff-eval

> DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction — Jiang et al. (2021) (arXiv:2108.09091, 2021)

## What this evaluates

Evaluates the predictive accuracy and computational efficiency of deep learning models for urban traffic forecasting. It benchmarks grid-based, graph-based, and multivariate time-series architectures on standard traffic datasets to compare their ability to capture spatiotemporal dependencies.

## Datasets

- **BikeNYC-I** — total ?; splits: train (-1), val (-1), test (-1)
- **TaxiNYC** — total ?; splits: train (-1), val (-1), test (-1)
- **TaxiBJ** — total ?; splits: train (-1), val (-1), test (-1)
- **METR-LA** — total ?; splits: train (-1), val (-1), test (-1)
- **PeMS-BAY** — total ?; splits: train (-1), val (-1), test (-1)
- **PEMSD7M** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error: the average of the absolute differences between predicted and actual values. Used as the default loss function during training.
- `RMSE` — range: other
  - Root Mean Square Error: the square root of the average of squared differences between predicted and actual values.
- `MAPE` — range: percent
  - Mean Absolute Percentage Error: the average of absolute percentage differences between predicted and actual values, expressed as a percentage.

## Input / output format

**Input**: Historical traffic measurements (volume or speed) over a fixed observation window, structured as grid cells or graph nodes with temporal dimensions.

**Output**: Predicted traffic measurements for a specified prediction horizon (1 step for grid models; 1, 2, or 3 steps for graph models).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    mask = y_true != 0
    y_t, y_p = y_true[mask], y_pred[mask]
    mae = np.mean(np.abs(y_t - y_p))
    rmse = np.sqrt(np.mean((y_t - y_p) ** 2))
    mape = np.mean(np.abs((y_t - y_p) / y_t)) * 100
    return {'MAE': mae, 'RMSE': rmse, 'MAPE': mape}
```

## Common pitfalls

- Zero values in the ground truth are explicitly ignored when computing MAE, RMSE, and MAPE.
- Grid-based and graph-based models use different fixed observation/prediction step settings (6 vs 12), so direct comparison requires matching these horizons.
- Training uses early stopping if validation error converges within 10 epochs, otherwise runs for a maximum of 200 epochs; the best validation model is saved.

## Evidence (verbatim from paper)

> Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE) are used as metrics, where zero values will be ignored.

## Citation

```bibtex
@misc{jiang2021dltraff,
  title={DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction},
  author={Jiang et al. (2021)},
  year={2021},
  note={arXiv:2108.09091}
}
```

- arXiv: 2108.09091