V2v Qa Eval

---
name: v2v-qa-eval
description: Evaluates a multi-modal LLM's ability to fuse 3D perception features from multiple connected vehicles to answer safety-critical driving queries. It probes spatial grounding, notable object identification near planned waypoints, and collision-avoidance trajectory planning in cooperative autonomous driving scenarios. Use when the user wants to benchmark on V2V-QA, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.09980
  bibtex_key: chiu2025v2vllm
  confidence: high
---

# v2v-qa-eval

> V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models — Chiu et al. (2025) (arXiv:2502.09980, 2025)

## What this evaluates

Evaluates a multi-modal LLM's ability to fuse 3D perception features from multiple connected vehicles to answer safety-critical driving queries. It probes spatial grounding, notable object identification near planned waypoints, and collision-avoidance trajectory planning in cooperative autonomous driving scenarios.

## Datasets

- **V2V-QA** — total ?; splits: V2V-split (-1), V2X-split (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for spatial grounding and object identification tasks. Computed per question type (Q1-Q4) and averaged for QGr.
- `L2 distance error` — range: other
  - Euclidean distance between predicted future trajectory waypoints and ground-truth trajectory waypoints, measured in meters.
- `Collision rate` — range: percent
  - Percentage of generated planning trajectories that result in a collision with nearby objects or obstacles.
- `Communication cost` — range: other
  - Total data transferred per timestep between CAVs and the centralized LLM node, calculated as (0.203 + 0.0004 * N_q) MB per CAV, where N_q is the number of questions.

## Input / output format

**Input**: Scene-level feature maps and object-level feature vectors extracted from LiDAR point clouds of multiple connected autonomous vehicles (CAVs), combined with a natural language question specifying grounding references, object identification targets, or planning waypoints.

**Output**: Natural language response containing predicted spatial coordinates for grounding, identified notable objects, and future driving trajectory waypoints.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # Grounding & Identification (Q1-Q4)
    f1_scores = []
    for pred_locs, gold_locs in zip(predictions, gold):
        tp = count_spatial_matches(pred_locs, gold_locs, threshold=0.5)
        prec = tp / (len(pred_locs) + 1e-8)
        rec = tp / (len(gold_locs) + 1e-8)
        f1_scores.append(2 * prec * rec / (prec + rec + 1e-8))
    f1 = mean(f1_scores)
    
    # Planning (Q5)
    l2_errors = [np.linalg.norm(p - g) for p, g in zip(predictions, gold)]
    l2 = mean(l2_errors)
    cr = count_collisions(predictions) / len(predictions)
    
    return {'F1': f1, 'L2': l2, 'CR': cr}
```

## Common pitfalls

- F1 scores for grounding tasks rely on implicit spatial matching thresholds not explicitly defined in the main text, making exact replication difficult without supplementary code.
- Planning metrics (L2 distance and collision rate) require specific trajectory alignment and collision detection logic that varies across implementations.
- Communication cost scales non-linearly with the number of vehicles (N_v) and questions (N_q), so direct comparisons are sensitive to these hyperparameters rather than reflecting pure model efficiency.

## Evidence (verbatim from paper)

> Q1: Grounding at a reference location. Q2: Grounding behind a reference object at a location. Q3: Grounding behind a reference object in a direction. QGr: Average of grounding (Q1, Q2, and Q3). Q4: Notable object identification. Q5: Planning. L2: L2 distance error. CR: Collision rate.

## Citation

```bibtex
@misc{chiu2025v2vllm,
  title={V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models},
  author={Chiu et al. (2025)},
  year={2025},
  note={arXiv:2502.09980}
}
```

- arXiv: 2502.09980