Can Qa Eval

---
name: can-qa-eval
description: Evaluates large language models' ability to perform structured reasoning over temporally segmented in-vehicle CAN traffic logs. It probes capabilities in temporal analysis, multi-condition inference, and behavioral interpretation for automotive cybersecurity forensics. Use when the user wants to benchmark on CAN-QA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.24935
  bibtex_key: chen2026canqa
  confidence: high
---

# can-qa-eval

> CAN-QA: A Question-Answering Benchmark for Reasoning over In-Vehicle CAN Traffic — Chen et al. (2026) (arXiv:2604.24935, 2026)

## What this evaluates

Evaluates large language models' ability to perform structured reasoning over temporally segmented in-vehicle CAN traffic logs. It probes capabilities in temporal analysis, multi-condition inference, and behavioral interpretation for automotive cybersecurity forensics.

## Datasets

- **CAN-QA** — total 33128; splits: test (-1); repo https://github.com/Kriiiiss/CAN-QA

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the proportion of correctly answered questions out of the total number of questions. Evaluated separately for True/False (TF) and Multiple-Choice (MCQ) formats, as well as across ten reasoning categories.

## Input / output format

**Input**: A temporally segmented window of in-vehicle CAN traffic logs paired with a natural-language question describing a specific traffic property or anomaly.

**Output**: For TF tasks: a binary 'True' or 'False' answer. For MCQ tasks: selection of the single most precise explanation from a set of multiple-choice options.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return correct / len(gold) * 100
```

## Common pitfalls

- Models often rely on coarse statistical cues (e.g., presence of many IDs) rather than verifying exact numerical thresholds, leading to errors in interval-based reasoning.
- MCQ tasks are significantly harder than TF tasks because models must discriminate between multiple plausible alternatives with subtle quantitative or structural differences, not just verify a single statement.

## Evidence (verbatim from paper)

> Figure 3 presents the zero-shot prediction accuracy of the selected LLMs on the TF and MCQ tasks. Across models, TF accuracy ranges from 47% to 59%, while MCQ accuracy ranges from 25% to 40%.

## Citation

```bibtex
@misc{chen2026canqa,
  title={CAN-QA: A Question-Answering Benchmark for Reasoning over In-Vehicle CAN Traffic},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2604.24935}
}
```

- arXiv: 2604.24935