R U Maad Eval

---
name: r-u-maad-eval
description: Evaluates a model's ability to perform unsupervised anomaly detection on multi-agent traffic trajectories in urban environments. It probes frame-wise recognition of rare and abnormal driving behaviors, including both individual maneuvers and context-dependent interactions between agents and static map features. Use when the user wants to benchmark on R-U-MAAD, or asks about evaluating this task. Reports frame-wise anomaly detection accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.01838
  bibtex_key: wiederer2022rumaad
  confidence: high
---

# r-u-maad-eval

> A Benchmark for Unsupervised Anomaly Detection in Multi-Agent Trajectories — Wiederer et al. (2022) (arXiv:2209.01838, 2022)

## What this evaluates

Evaluates a model's ability to perform unsupervised anomaly detection on multi-agent traffic trajectories in urban environments. It probes frame-wise recognition of rare and abnormal driving behaviors, including both individual maneuvers and context-dependent interactions between agents and static map features.

## Datasets

- **R-U-MAAD** — total 245574; splits: train (205942), val (39472), test (160); repo https://github.com/againerju/r_u_maad

## Metrics

- `frame-wise anomaly detection accuracy` **(primary)** — range: [0, 1]
  - Binary classification accuracy computed per time step (normal vs abnormal). Ignore regions corresponding to transition phases are explicitly excluded from the calculation, ensuring errors in those steps do not affect the final score.

## Input / output format

**Input**: Multi-agent trajectory sequences represented as 2D birds-eye view center coordinates sampled at 10 Hz. Models receive 2 seconds of historical trajectory data to evaluate or predict the next 3 seconds.

**Output**: Per time step: binary anomaly label (normal or abnormal) or anomaly score. Predictions must align with the frame-wise annotation timeline and exclude ignore regions.

## Scoring recipe

```python
def compute_frame_wise_accuracy(predictions, gold_labels, ignore_mask):
    valid_preds = [p for p, ign in zip(predictions, ignore_mask) if not ign]
    valid_gold = [g for g, ign in zip(gold_labels, ignore_mask) if not ign]
    if len(valid_gold) == 0:
        return 0.0
    correct = sum(1 for p, g in zip(valid_preds, valid_gold) if p == g)
    return correct / len(valid_gold)
```

## Common pitfalls

- Including 'ignore regions' (transition phases) in the evaluation, which the protocol explicitly excludes from the calculation.
- Assuming the test set is purely simulated; it is a hybrid where only one agent is animated in simulation while others are replayed from real Argoverse data.
- Treating all 13 anomaly classes uniformly without considering the actor-interactive vs. map-interactive distinctions, which significantly impacts algorithm design and failure analysis.

## Evidence (verbatim from paper)

> We define abnormal driving as the set of all uncommon and rare scenarios and provide a test set with frame-wise annotations of diverse driving anomalies... Ignore regions are not considered during evaluation, such that errors in ignored time steps do not contribute to the result.

## Citation

```bibtex
@misc{wiederer2022rumaad,
  title={A Benchmark for Unsupervised Anomaly Detection in Multi-Agent Trajectories},
  author={Wiederer et al. (2022)},
  year={2022},
  note={arXiv:2209.01838}
}
```

- arXiv: 2209.01838