Nexar Collision Eval

---
name: nexar-collision-eval
description: Evaluates autonomous ML research agents on their ability to search a mixed categorical-continuous configuration space for optimal model architectures and training setups. It measures convergence speed and final predictive performance on a binary collision prediction task using pre-extracted dashcam video features. Use when the user wants to benchmark on Nexar dashcam collision prediction dataset, or asks about evaluating this task. Reports AP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15916
  bibtex_key: li2026autoresearching
  confidence: high
---

# nexar-collision-eval

> Auto Researching, not hyperparameter tuning: Convergence Analysis of 10,000 Experiments — Li (2026) (arXiv:2603.15916, 2026)

## What this evaluates

Evaluates autonomous ML research agents on their ability to search a mixed categorical-continuous configuration space for optimal model architectures and training setups. It measures convergence speed and final predictive performance on a binary collision prediction task using pre-extracted dashcam video features.

## Datasets

- **Nexar dashcam collision prediction dataset** — total 1500; splits: train (1200), val (150), test (150)

## Metrics

- `AP` **(primary)** — range: [0, 1]
  - Average Precision for binary classification, computed as the area under the precision-recall curve. Standard convention for threshold-agnostic evaluation.
- `convergence_exponent_c` — range: other
  - Exponent c in the power-law model AP*(N) = a - b * N^(-c), fitted to the cumulative best AP over N experiments. Larger c indicates faster convergence.

## Input / output format

**Input**: Natural language text encoding of the experiment history H_{t-1}, including past configurations, validation AP scores, failure diagnostics, and idea genealogy, plus the task description.

**Output**: Structured YAML configuration specifying discrete choices (backbone, encoder, pooling, loss type, scheduler) and continuous/hyperparameter values (learning rate, weight decay, batch size, sequence length, epochs, oversampling ratio, mixup alpha, feature noise sigma).

## Scoring recipe

```python
def compute_ap(y_true, y_prob):
    precisions, recalls, _ = precision_recall_curve(y_true, y_prob)
    return np.trapz(precisions, recalls)
# Applied to validation set for search guidance; final AP reported on held-out test set when available.
```

## Common pitfalls

- Validation AP is used for search guidance, but direct leaderboard comparison requires the held-out competition test set which may not be fully evaluated.
- The search space is mixed categorical-continuous; comparing LLM search directly to purely numeric Bayesian methods (TPE) without accounting for architectural reasoning advantages is misleading.
- Performance variance is dominated by architectural choices (94%) rather than hyperparameter tuning (6%), so focusing solely on LR/batch size optimization misses the primary driver of success.

## Evidence (verbatim from paper)

> The Nexar collision prediction challenge is an active Kaggle competition; our 0.9245 validation AP provides a strong baseline, though direct leaderboard comparison requires evaluation on the held-out competition test set (which we report when available). We use focal loss to down-weight easy examples and AP as the primary metric following the original challenge evaluation.

## Citation

```bibtex
@misc{li2026autoresearching,
  title={Auto Researching, not hyperparameter tuning: Convergence Analysis of 10,000 Experiments},
  author={Li (2026)},
  year={2026},
  note={arXiv:2603.15916}
}
```

- arXiv: 2603.15916