Scenicrules Eval

---
name: scenicrules-eval
description: Evaluates autonomous driving agents on their ability to navigate stochastic traffic scenarios while satisfying a hierarchical set of multi-objective specifications. It probes how well agents balance conflicting goals like collision avoidance, road compliance, passenger comfort, and progress under varying priority constraints. Use when the user wants to benchmark on ScenicRules Benchmark, or asks about evaluating this task. Reports Violation Score (VS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.16073
  bibtex_key: chang2026scenicrules
  confidence: high
---

# scenicrules-eval

> ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios — Chang et al. (2026) (arXiv:2602.16073, 2026)

## What this evaluates

Evaluates autonomous driving agents on their ability to navigate stochastic traffic scenarios while satisfying a hierarchical set of multi-objective specifications. It probes how well agents balance conflicting goals like collision avoidance, road compliance, passenger comfort, and progress under varying priority constraints.

## Datasets

- **ScenicRules Benchmark** — total ?; splits: test (-1); repo https://github.com/BerkeleyLearnVerify/ScenicRules

## Metrics

- `Violation Score (VS)` **(primary)** — range: other
  - Quantifies the extent of violation for each of the 19 formalized rules. Computed using Signal Temporal Logic (STL) robustness metrics or custom objective functions over the trajectory. A positive value indicates a violation, with larger values denoting greater severity.

## Input / output format

**Input**: Simulation trajectory data including ego and agent positions (polygons), velocities, accelerations, and environmental maps (drivable areas, lane boundaries, target regions) over the simulation time interval [T1, T2].

**Output**: A violation score (VS) value for each evaluated rule, or a dictionary mapping rule IDs to their respective VS values.

## Scoring recipe

```python
def compute_vs(traj, env, rule_id):
    # Extract trajectory states and environment parameters
    p_ego, v_ego, a_ego = traj.positions, traj.velocities, traj.accelerations
    vs = 0.0
    if rule_id == 3:  # Staying within drivable area
        out_area = max_t(area(p_ego(t) \ env.drivable_area))
        dist_term = max_t(dist(p_ego(t), env.drivable_area))
        vs = out_area + dist_term**2
    elif rule_id == 13:  # Speed limit
        vs = max(max_t(norm(v_ego(t)) - env.speed_limit), 0)**2
    elif rule_id == 15:  # Lane centering
        vs = sum_t(dist(centroid(p_ego(t)), env.centerline(t)))
    # ... other rules follow similar pattern using STL robustness or objective functions
    return vs
```

## Common pitfalls

- Using only STL robustness without distance-based terms may fail to differentiate violation severity once an agent is fully outside a boundary.
- Default priority groupings are reference baselines; actual evaluation requires context-specific priority permutations (e.g., emergency vs. dense traffic).
- Choosing between max-vs-sum violation aggregation changes which trajectories are considered optimal.

## Evidence (verbatim from paper)

> Each rule is defined by a logical formula together with a corresponding violation score (VS) that measures the extent of violation. A positive VS indicates a violation, and larger values correspond to more severe violations.

## Citation

```bibtex
@misc{chang2026scenicrules,
  title={ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios},
  author={Chang et al. (2026)},
  year={2026},
  note={arXiv:2602.16073}
}
```

- arXiv: 2602.16073