Drivecritic Eval

---
name: drivecritic-eval
description: Evaluates a model's ability to judge autonomous driving trajectory pairs based on context-aware reasoning, safety, and human preferences, rather than relying on rigid rule-based thresholds. It probes whether the model can integrate visual and symbolic context to reason about nuanced traffic situations like lateral buffer maintenance or stop sign compliance. Use when the user wants to benchmark on DriveCritic, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.13108
  bibtex_key: song2025drivecritic
  confidence: high
---

# drivecritic-eval

> DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models — Jingyu Song et al. (2025) (arXiv:2510.13108, 2025)

## What this evaluates

Evaluates a model's ability to judge autonomous driving trajectory pairs based on context-aware reasoning, safety, and human preferences, rather than relying on rigid rule-based thresholds. It probes whether the model can integrate visual and symbolic context to reason about nuanced traffic situations like lateral buffer maintenance or stop sign compliance.

## Datasets

- **DriveCritic** — total ?; splits: train (1100), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of pairwise comparisons where the model's predicted preferred trajectory matches the human-preferred trajectory.
- `Robustness Rate (RR)` — range: [0, 1]
  - Fraction of test pairs where the model's prediction remains unchanged after swapping the order of the two trajectories in the prompt: RR = (1/|D|) * sum(I[y^i == y_hat^i]).

## Input / output format

**Input**: A prompt containing visual context (stitched camera images and BEV maps), ego vehicle status, and two candidate trajectory waypoints (Trajectory A and Trajectory B) for a 4-second horizon.

**Output**: A single judgment indicating which of the two trajectories (A or B) is preferred by the model.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    return correct / len(predictions)

def compute_robustness_rate(original_preds, flipped_preds):
    consistent = sum(1 for p, fp in zip(original_preds, flipped_preds) if p == fp)
    return consistent / len(original_preds)
```

## Common pitfalls

- Models often exhibit position bias, preferring the first trajectory listed in the prompt regardless of actual quality.
- Rule-based metrics like EPDMS rely on fixed thresholds and fail to capture nuanced social norms or safety trade-offs in ambiguous scenarios.

## Evidence (verbatim from paper)

> The primary evaluation metric is *accuracy*, defined as the proportion of pairwise comparisons in which the model’s judgment agrees with the human-preferred trajectory. ... To quantify robustness, we perform a *position-flip test*: for every test pair, we swap the order of Trajectory A and Trajectory B in the prompt and re-evaluate the model. We follow[[36]] to compute the *Robustness Rate* (RR): RR = (1/|D|)∑ I[y^i = ŷ^i]

## Citation

```bibtex
@misc{song2025drivecritic,
  title={DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models},
  author={Jingyu Song et al. (2025)},
  year={2025},
  note={arXiv:2510.13108}
}
```

- arXiv: 2510.13108