Simbarca Traffic Forecasting Eval

---
name: simbarca-traffic-forecasting-eval
description: Evaluates the ability of deep learning models to forecast urban traffic speeds at both the individual road segment and regional levels. It probes spatio-temporal forecasting capabilities under varying congestion levels, testing how well models integrate multi-source sensor data (drone trajectories and loop detectors) to predict future traffic states. Use when the user wants to benchmark on SimBarca, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.03492
  bibtex_key: xiong2025simbarca
  confidence: high
---

# simbarca-traffic-forecasting-eval

> Multi-Source Urban Traffic Flow Forecasting with Drone and Loop Detector Data — Xiong et al. (2025) (arXiv:2501.03492, 2025)

## What this evaluates

Evaluates the ability of deep learning models to forecast urban traffic speeds at both the individual road segment and regional levels. It probes spatio-temporal forecasting capabilities under varying congestion levels, testing how well models integrate multi-source sensor data (drone trajectories and loop detectors) to predict future traffic states.

## Datasets

- **SimBarca** — total 101; splits: train (75), test (26)

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error: average of absolute differences between predicted and true speeds across all time steps and segments. Formula: MAE = (1/n) Σ |ŷ_i - y_i|.
- `RMSE` — range: other
  - Root Mean Square Error: square root of the average of squared differences between predicted and true speeds. Formula: RMSE = sqrt((1/n) Σ (ŷ_i - y_i)^2).
- `MAPE*` — range: percent
  - Modified Mean Absolute Percentage Error: average of absolute percentage errors, computed only where true speed > 1 m/s to avoid division by zero. Formula: MAPE* = (1/n) Σ |(ŷ_i - y_i)/y_i| for y_i > 1 m/s.

## Input / output format

**Input**: Time-series sequences of road segment speeds (from drones, sampled every 5s) and point speeds (from loop detectors, aggregated to 3-min intervals) over a 30-minute historical window, combined with an undirected graph representation of the road network (1570 nodes, 2803 edges).

**Output**: Predicted future traffic speeds (in m/s) for specific road segments and aggregated spatial regions over 15-minute or 30-minute horizons (5 or 10 time steps).

## Scoring recipe

```python
def compute_metrics(pred, true):
    # pred, true: arrays of shape (num_segments, num_steps)
    # Metrics are calculated per segment/region, then averaged
    mae = np.mean(np.abs(pred - true))
    rmse = np.sqrt(np.mean((pred - true)**2))
    mask = true > 1.0
    mape_star = np.mean(np.abs((pred[mask] - true[mask]) / true[mask])) * 100
    return mae, rmse, mape_star
```

## Common pitfalls

- MAPE becomes infinite when true speed is 0; the paper explicitly uses MAPE* by filtering out samples where speed ≤ 1 m/s.
- Loop detector data (point speed) systematically overestimates true segment speed due to sensor placement relative to stop lines, which can bias single-modality baselines if not accounted for.
- Prediction horizons are fixed at 15 and 30 minutes, corresponding to 5 and 10 discrete time steps in the dataset, not continuous time intervals.

## Evidence (verbatim from paper)

> Following the common practice in traffic forecasting literature[[8]], the prediction results are evaluated with three metrics: MAE, Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE). Their vector forms for a pair of prediction ($\hat{\mathbf{y}}$) and ground truth ($\mathbf{y}$) are defined as follows: [formula] where $n$ is an index for flattened predictions and labels. In the evaluation, the speed values take the unit of m/s, and the metrics are calculated for each road segment (or region) and then averaged over all segments (regions). Since a zero segment speed in the label will result in infinite MAPE, we only evaluate MAPE when the speed value is greater than 1 m/s, and we refer to this modified metric as MAPE*.

## Citation

```bibtex
@misc{xiong2025simbarca,
  title={Multi-Source Urban Traffic Flow Forecasting with Drone and Loop Detector Data},
  author={Xiong et al. (2025)},
  year={2025},
  note={arXiv:2501.03492}
}
```

- arXiv: 2501.03492