Talk2car Eval

---
name: talk2car-eval
description: Evaluates a model's ability to ground free-form natural language commands to specific objects in autonomous driving scenes. It probes spatial and relational language understanding, disambiguation of same-category objects, and handling of long-range referents and complex sentences. Use when the user wants to benchmark on Talk2Car, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1909.10838
  bibtex_key: deruyttere2019talk2car
  confidence: medium
---

# talk2car-eval

> Talk2Car: Taking Control of Your Self-Driving Car — Deruyttere et al. (2019) (arXiv:1909.10838, 2019)

## What this evaluates

Evaluates a model's ability to ground free-form natural language commands to specific objects in autonomous driving scenes. It probes spatial and relational language understanding, disambiguation of same-category objects, and handling of long-range referents and complex sentences.

## Datasets

- **Talk2Car** — total 11959; splits: train (8349), val (1163), test (2447)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly identified referred objects. (Exact formula not specified in the provided dataset section; standard for object referral tasks.)

## Input / output format

**Input**: A video frame (or short sequence) from an autonomous vehicle's front-facing camera, accompanied by a free-form natural language command referring to a specific object in the scene.

**Output**: Identification of the referred object, typically as a bounding box or object class/ID from the nuScenes annotation schema.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_obj in zip(predictions, gold):
        if pred == gold_obj:
            correct += 1
    return correct / len(gold)
# Note: Exact scoring protocol not detailed in the provided text.
```

## Common pitfalls

- Samples from the same video are strictly kept within a single split to prevent temporal data leakage.
- The test set is not a single monolithic set; it includes curated subsets for specific conditions (e.g., object distance, command length, category occurrence counts) to evaluate targeted capabilities.
- nuScenes test set 3D bounding boxes are undisclosed, so the Talk2Car test set relies on nuScenes training set annotations, which may affect generalization evaluation.

## Evidence (verbatim from paper)

> The Talk2Car dataset evaluates state-of-the-art models on disambiguating objects via spatial and relational language cues, handling complex sentences, distant objects, and real-time prediction speed—key challenges in real-world AV interactions. We have also identified multiple subsets of the test set, which allow evaluation of specific situations.

## Citation

```bibtex
@misc{deruyttere2019talk2car,
  title={Talk2Car: Taking Control of Your Self-Driving Car},
  author={Deruyttere et al. (2019)},
  year={2019},
  note={arXiv:1909.10838}
}
```

- arXiv: 1909.10838