Tb Bench Eval

---
name: tb-bench-eval
description: This benchmark evaluates multi-modal large language models' ability to understand spatio-temporal traffic behaviors from ego-centric dashcam images and videos. It probes eight distinct perception tasks, including road detection, object-lane alignment, turning prediction, and ego-trajectory estimation, requiring both spatial reasoning and temporal tracking. Use when the user wants to benchmark on TB-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.05733
  bibtex_key: charoenpitaks2025tbbench
  confidence: high
---

# tb-bench-eval

> TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos — Charoenpitaks et al. (2025) (arXiv:2501.05733, 2025)

## What this evaluates

This benchmark evaluates multi-modal large language models' ability to understand spatio-temporal traffic behaviors from ego-centric dashcam images and videos. It probes eight distinct perception tasks, including road detection, object-lane alignment, turning prediction, and ego-trajectory estimation, requiring both spatial reasoning and temporal tracking.

## Datasets

- **TB-Bench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers across all instances. Calculated as (number of correct predictions / total number of predictions) * 100. Reported per task and averaged across all eight tasks.

## Input / output format

**Input**: A sequence of N dashcam frames (N=8 for temporal tasks, N=1 otherwise), each resized to 384×384 pixels and processed into 16 visual embeddings per frame, paired with a text prompt containing a question and, for classification tasks, a list of multiple-choice options.

**Output**: A text string containing either the selected class label from the provided options or a numerical value formatted as specified in the prompt (e.g., 'xx.x meters').

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_clean = pred.strip().lower()
        gold_clean = gold.strip().lower()
        if pred_clean == gold_clean or gold_clean in pred_clean:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- In-context learning uses different numbers of examples depending on the task type: 3 examples for single-frame tasks vs. 1 example for multi-frame tasks.
- Numerical output questions require strict adherence to the prompt's formatting instruction (e.g., 'Answer in xx.x meters') to avoid parsing failures during evaluation.
- The random guess baseline is reported as 0.0% accuracy rather than the theoretical chance level, which can skew comparative analysis if not accounted for.

## Evidence (verbatim from paper)

> Table 4: Results of compared methods on TB-Bench are reported in accuracy (%), where higher indicates better performance. Random guess† results are considered zero. ⋆In-context learning for single-frame tasks uses three in-context examples, while multi-frame tasks use one.

## Citation

```bibtex
@misc{charoenpitaks2025tbbench,
  title={TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos},
  author={Charoenpitaks et al. (2025)},
  year={2025},
  note={arXiv:2501.05733}
}
```

- arXiv: 2501.05733