Ozone Eval

---
name: ozone-eval
description: Evaluates a unified platform for standardizing heterogeneous transportation trajectory data, automating cross-dataset conversion, and benchmarking safety and behavior models across multiple cities and datasets. Use when the user wants to benchmark on Ozone Standardized Trajectory Suite (NGSIM, highD, CitySim, UTE), or asks about evaluating this task. Reports cross-city F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.10959
  bibtex_key: zheng2026ozone
  confidence: high
---

# ozone-eval

> Ozone: A Unified Platform for Transportation Research — Ou Zheng et al. (2026) (arXiv:2604.10959, 2026)

## What this evaluates

Evaluates a unified platform for standardizing heterogeneous transportation trajectory data, automating cross-dataset conversion, and benchmarking safety and behavior models across multiple cities and datasets.

## Datasets

- **Ozone Standardized Trajectory Suite (NGSIM, highD, CitySim, UTE)** — total ?; splits: train (-1), test (-1)

## Metrics

- `cross-city F1 score` **(primary)** — range: [0, 1]
  - F1 score for conflict-type classification and AUC for crash-risk prediction, computed on held-out city data. Transfer efficiency is calculated as the ratio of cross-city performance to within-city performance.
- `conversion error` — range: meters, degrees, m/s
  - Mean absolute error between converted trajectory fields (position, heading, speed) and ground-truth annotations.
- `reproducibility variance` — range: percent
  - Percentage discrepancy between reproduced results and originally published numbers across standardized vs. ad-hoc preprocessing pipelines.

## Input / output format

**Input**: Raw trajectory data (position, heading, speed, acceleration, OBB corners) from heterogeneous datasets (NGSIM, highD, CitySim, UTE) and digital-twin maps.

**Output**: Standardized schema fields, computed safety indicators (TTC, PET, etc.), and model predictions (conflict-type classification, crash-risk probability).

## Scoring recipe

```python
def evaluate(predictions, gold, config):
    # 1. Data conversion accuracy
    pos_err = mean(abs(predictions.pos - gold.pos))
    head_err = mean(abs(predictions.heading - gold.heading))
    speed_err = mean(abs(predictions.speed - gold.speed))
    
    # 2. Safety metrics (TTC, PET, etc.) computed via standardized pipeline
    safety_scores = compute_safety_indicators(predictions, gold)
    
    # 3. Model performance (F1/AUC)
    f1 = f1_score(gold.labels, predictions.conflict_type)
    auc = roc_auc_score(gold.labels, predictions.risk_prob)
    
    # 4. Reproducibility check
    variance = abs(reproduced_result - published_result) / published_result
    
    return {'pos_err': pos_err, 'head_err': head_err, 'speed_err': speed_err,
            'f1': f1, 'auc': auc, 'variance': variance}
```

## Common pitfalls

- Ad-hoc preprocessing and undocumented code cause high variability in baseline results (8-22% discrepancies).
- Dataset-specific coordinate systems, formats, and naming conventions must be explicitly normalized before cross-city evaluation.
- Safety metric thresholds (e.g., TTC, PET) must be consistently defined across datasets to ensure comparability.

## Evidence (verbatim from paper)

> The average cross-city F1 score for conflict-type classification was 0.83, compared with 0.91 for within-city evaluation, corresponding to a transfer efficiency of 91%. For crash-risk prediction, the cross-city AUC was 0.79, compared with 0.86 for within-city evaluation (92% transfer efficiency).

## Citation

```bibtex
@misc{zheng2026ozone,
  title={Ozone: A Unified Platform for Transportation Research},
  author={Ou Zheng et al. (2026)},
  year={2026},
  note={arXiv:2604.10959}
}
```

- arXiv: 2604.10959