Roundabout Tau Eval

---
name: roundabout-tau-eval
description: This benchmark evaluates a model's ability to detect traffic anomalies in roadside surveillance videos and generate detailed, reasoning-grounded textual summaries of those anomalies. It probes both binary/fine-grained classification accuracy and semantic alignment of generated descriptions with ground-truth event narratives. Use when the user wants to benchmark on Roundabout-TAU, or asks about evaluating this task. Reports 4-cls AP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.19098
  bibtex_key: lin2026tau_r1
  confidence: high
---

# roundabout-tau-eval

> TAU-R1: Visual Language Model for Traffic Anomaly Understanding — Lin et al. (2026) (arXiv:2603.19098, 2026)

## What this evaluates

This benchmark evaluates a model's ability to detect traffic anomalies in roadside surveillance videos and generate detailed, reasoning-grounded textual summaries of those anomalies. It probes both binary/fine-grained classification accuracy and semantic alignment of generated descriptions with ground-truth event narratives.

## Datasets

- **Roundabout-TAU** — total 342; splits: train (300), test (42); repo https://github.com/siri-rouser/TAU-R1

## Metrics

- `4-cls AP` **(primary)** — range: [0, 1]
  - Average Precision across four traffic anomaly classes, computed from predicted class probabilities against ground-truth labels.
- `4-cls F1` — range: [0, 1]
  - Macro-averaged F1 score across four traffic anomaly classes.
- `2-cls AP` — range: [0, 1]
  - Average Precision for binary anomaly detection (anomalous vs. normal).
- `2-cls F1` — range: [0, 1]
  - F1 score for binary anomaly detection.
- `BLEU` — range: [0, 1]
  - Standard n-gram overlap metric for text generation, typically BLEU-4.
- `METEOR` — range: [0, 1]
  - Metric evaluating alignment between generated and reference text using synonymy, stemming, and exact matches.
- `ROUGE-L` — range: [0, 1]
  - Recall-Oriented Understudy for Gisting Evaluation based on longest common subsequence.
- `G-Score` — range: [0, 10]
  - LLM-judged score (0–10) summing four aspects: environment correctness (0–1), object grounding (0–2), description quality (0–5), and reasoning quality (0–2).

## Input / output format

**Input**: Video clips (frames/timestamps) for classification; video clips paired with anomaly event prompts/questions for summarization.

**Output**: Classification: predicted class label (4-class or binary). Summarization: free-text description/summary of the anomalous event.

## Scoring recipe

```python
def score_classification(preds, gold):
    ap_4cls = compute_average_precision(preds['4cls'], gold['4cls'])
    f1_4cls = compute_f1(preds['4cls'], gold['4cls'])
    ap_2cls = compute_average_precision(preds['2cls'], gold['2cls'])
    f1_2cls = compute_f1(preds['2cls'], gold['2cls'])
    return ap_4cls, f1_4cls, ap_2cls, f1_2cls

def score_summarization(pred_text, ref_text):
    bleu = compute_bleu(pred_text, ref_text)
    meteor = compute_meteor(pred_text, ref_text)
    rouge_l = compute_rouge_l(pred_text, ref_text)
    aspects = llm_judge_score(pred_text, ref_text, prompt_template)
    g_score = aspects.env_correctness + aspects.object_grounding + aspects.desc_quality + aspects.reasoning_quality
    return bleu, meteor, rouge_l, g_score
```

## Common pitfalls

- Summarization metrics are only computed on the 34 anomalous test videos, not all 42 test videos.
- G-Score relies on an external LLM judge (ChatGPT-5) with specific prompt templates provided only in supplementary material, making exact replication difficult.
- Test set is small (42 videos) and class distribution is maintained from the full dataset, which may lead to high variance in AP/F1 scores.

## Evidence (verbatim from paper)

> We evaluate TAU-R1 on the two tasks defined in Roundabout-TAU: anomaly classification and anomaly summarization. For classification, we report both four-class and binary classification performance using Average Precision (AP) and F1 score. For summarization, following prior video anomaly understanding work [zhang2025holmes, xing2025echotraffic], we use standard text-generation metrics including BLEU [papineni2002bleu], METEOR [banerjee2005meteor] and ROUGE-L [lin2004rouge]. In addition to those traditional text-based method, we further introduce GPT-Eval to better assess semantic accuracy and reasoning quality, following recent VLM evaluation practice [liu2023visual, tang2024hawk]. Specifically, we prompt ChatGPT-5 to score each prediction from 0 to 10 from four aspects: environment correctness (0–1), object grounding (0–2), description quality (0–5), and reasoning quality (0–2).

## Citation

```bibtex
@misc{lin2026tau_r1,
  title={TAU-R1: Visual Language Model for Traffic Anomaly Understanding},
  author={Lin et al. (2026)},
  year={2026},
  note={arXiv:2603.19098}
}
```

- arXiv: 2603.19098