Car Bench Eval

---
name: car-bench-eval
description: Evaluates LLM agents' ability to resolve uncertainty and adhere to safety policies in automotive environments. It probes limit-awareness, consistency across multiple attempts, and robust multi-turn tool use under incomplete or ambiguous user requests. Use when the user wants to benchmark on CAR-bench, or asks about evaluating this task. Reports Passˆ3.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.22027
  bibtex_key: kirmayr2026carbench
  confidence: high
---

# car-bench-eval

> CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty — Kirmayr et al. (2026) (arXiv:2601.22027, 2026)

## What this evaluates

Evaluates LLM agents' ability to resolve uncertainty and adhere to safety policies in automotive environments. It probes limit-awareness, consistency across multiple attempts, and robust multi-turn tool use under incomplete or ambiguous user requests.

## Datasets

- **CAR-bench** — total 240; splits: test (240); repo https://github.com/CAR-bench/car-bench

## Metrics

- `Passˆ3` **(primary)** — range: [0, 1]
  - The fraction of tasks where the model produces a correct solution in all k attempts (k=3), averaged across task types. It measures consistency rather than single-attempt success.

## Input / output format

**Input**: Multi-turn conversational prompts simulating automotive user requests, including ambiguous or incomplete scenarios, requiring tool use and policy adherence.

**Output**: Sequential tool calls (e.g., get operations, actions) and final responses, evaluated for correctness, policy compliance, and successful task completion.

## Scoring recipe

```python
def compute_pass_hat_k(predictions, gold, k=3):
    consistent_passes = 0
    for task_id, attempts in predictions.items():
        if all(check_pass(attempt, gold[task_id]) for attempt in attempts[:k]):
            consistent_passes += 1
    return consistent_passes / len(gold)
```

## Common pitfalls

- Confusing Pass@k (success in at least one of k attempts) with Passˆk (success in all k attempts), which measures consistency.
- Overlooking policy violations or tool execution failures when determining if a task 'passes', as the benchmark explicitly tracks these as distinct error modes.
- Failing to account for user-simulation errors, which can artificially lower Passˆk scores if the simulated user provides incorrect or hallucinated information.

## Evidence (verbatim from paper)

> We report Passˆ3 averaged across task types as our primary metric, prioritizing consistency in this safety-critical automotive domain. We set k=3 as it effectively discriminates model reliability while avoiding score saturation observed at higher k.

## Citation

```bibtex
@misc{kirmayr2026carbench,
  title={CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty},
  author={Kirmayr et al. (2026)},
  year={2026},
  note={arXiv:2601.22027}
}
```

- arXiv: 2601.22027