Cosmos Drive Dreams Eval

---
name: cosmos-drive-dreams-eval
description: Evaluates the effectiveness of a synthetic driving data generation pipeline by measuring performance gains in downstream autonomous driving perception tasks, including 3D lane detection, 3D object detection, and LiDAR-based detection, particularly under challenging conditions like extreme weather and nighttime. Use when the user wants to benchmark on Waymo Open Dataset, RDS-HQ, RDS-HQ-HL, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.09042
  bibtex_key: [REDACTED]
  confidence: high
---

# cosmos-drive-dreams-eval

> Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models — Ren et al. (2025) (arXiv:2506.09042, 2025)

## What this evaluates

Evaluates the effectiveness of a synthetic driving data generation pipeline by measuring performance gains in downstream autonomous driving perception tasks, including 3D lane detection, 3D object detection, and LiDAR-based detection, particularly under challenging conditions like extreme weather and nighttime.

## Datasets

- **Waymo Open Dataset** — total ?; splits: train (504), test (144)
- **RDS-HQ** — total ?; splits: train (-1), test (2800)
- **RDS-HQ-HL** — total 12000; splits: train (10000), test (2000)

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for 3D lane detection, computed per weather/time split.
- `Category Accuracy` — range: [0, 1]
  - Percentage of correctly classified lane categories.
- `LET-AP` — range: [0, 1]
  - Average Precision metric adapted for LiDAR/BEV object detection, evaluated at varying IoU thresholds for the vehicle category.
- `mAP` — range: [0, 1]
  - Mean Average Precision across object categories for LiDAR-based 3D object detection.

## Input / output format

**Input**: Rectified camera images (960x540 to 960x720), HDMap annotations, and optionally LiDAR point clouds (20 uniformly sampled frames per clip).

**Output**: Predicted 3D lane coordinates/segments, 3D bounding boxes for vehicles, or LiDAR point cloud detections.

## Scoring recipe

```python
def compute_f1(predictions, golds, iou_thresh=0.5):
    tp = sum(1 for p, g in zip(predictions, golds) if match(p, g, iou_thresh))
    fp = len(predictions) - tp
    fn = len(golds) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_map(predictions, golds, iou_thresh=0.5):
    scores, preds, golds = sort_by_confidence(predictions, golds)
    tp, fp = 0, 0
    for s, p, g in zip(scores, preds, golds):
        if match(p, g, iou_thresh): tp += 1
        else: fp += 1
    return tp / (tp + fp) if (tp + fp) > 0 else 0
```

## Common pitfalls

- The synthetic-to-real ratio ($R_{s2r}$) controls data mixing per training epoch, not the total dataset size.
- Corner-case splits (extreme weather/night) are curated subsets of the test set, not independent datasets.
- Metrics are reported per weather/time split, requiring careful aggregation rather than a single global score.

## Evidence (verbatim from paper)

> We report the F1-score and category accuracy of 3D lane detection trained on the Waymo Open Dataset and RDS-HQ (2k) in Tab. 1. The results show that Cosmos-Drive-Dreams significantly improves detection performance in cases where Albumumentations provide limited benefit.

## Citation

```bibtex
@misc{ren2025cosmosdrivedreams,
  title={Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models},
  author={Ren et al. (2025)},
  year={2025},
  note={arXiv:2506.09042}
}
```

- arXiv: 2506.09042