Bis Eval

---
name: bis-eval
description: Evaluates the trade-off between safety and efficiency of energy-function-based safe control algorithms in human-robot interaction and robot co-working scenarios. It tests how well different controllers navigate toward goals while avoiding collisions with human or robot agents across varying dynamic models. Use when the user wants to benchmark on BIS (Benchmark of Interactive Safety), or asks about evaluating this task. Reports efficiency_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1908.01883
  bibtex_key: wei2019safecontrol
  confidence: high
---

# bis-eval

> Safe Control Algorithms Using Energy Functions: A Unified Framework, Benchmark, and New Directions — Wei et al. (2019) (arXiv:1908.01883, 2019)

## What this evaluates

Evaluates the trade-off between safety and efficiency of energy-function-based safe control algorithms in human-robot interaction and robot co-working scenarios. It tests how well different controllers navigate toward goals while avoiding collisions with human or robot agents across varying dynamic models.

## Datasets

- **BIS (Benchmark of Interactive Safety)** — total 40; splits: test (40); repo https://github.com/intelligent-control-lab/BIS

## Metrics

- `efficiency_score` **(primary)** — range: other
  - Average number of goals achieved by the robot within the 30-second simulation period.
- `safety_score` — range: other
  - safety = -∑_{0}^{T} min(0, log(d/d_s)) * ḋ, where d is distance to obstacle, d_s is a threshold, and ḋ is relative velocity. Penalizes rapid approach to obstacles even without collision.
- `hybrid_score` — range: other
  - Maximum efficiency (goals achieved) achieved by the algorithm under the constraint of zero collisions. Used specifically for robot-robot collaboration scenarios.

## Input / output format

**Input**: Robot state, environment state, goal positions, and obstacle positions. The control module receives these to update state via Kalman filter, compute minimum distance/closest point, and generate control input.

**Output**: Control input (velocity/acceleration) applied to the robot simulator each frame at 20 fps.

## Scoring recipe

```python
def compute_metrics(goals_achieved, collisions, distances, velocities, T, d_s):
    efficiency = goals_achieved
    safety = 0.0
    for t in range(T):
        d = distances[t]
        dd = velocities[t]
        if d < d_s:
            safety += -min(0, math.log(d / d_s)) * dd
    hybrid = goals_achieved if collisions == 0 else 0
    return efficiency, safety, hybrid
```

## Common pitfalls

- The safety score includes a 'psychological safety' component that penalizes high relative velocity near obstacles even without physical collision, which differs from standard collision-rate metrics.
- The hybrid score is only applicable to robot-robot collaboration scenarios, not human-robot interactions, and strictly requires zero collisions to be valid.
- Performance is highly sensitive to algorithm-specific parameters (e.g., d_min, k, c1, c2, eta, lambda) which must be tuned per algorithm for fair comparison.

## Evidence (verbatim from paper)

> We use 40 pieces of 30 seconds long test scenarios to test different algorithms in the experiments. The frame rate is 20 fps. Three metrics are used to evaluate the performance of different algorithms: an efficiency score and a safety score for human-robot interactions, and a hybrid score for robot co-working. For all scores, the higher, the better. We use the average number of goals achieved in a given period by the robot as the efficiency score. Based on these considerations, the safety score is defined as safety = -∑_{0}^{T} min(0, log(d/d_s)) ḋ. We define the hybrid score as the maximum efficiency without collision to evaluate the performance of the algorithms in these situations.

## Citation

```bibtex
@misc{wei2019safecontrol,
  title={Safe Control Algorithms Using Energy Functions: A Unified Framework, Benchmark, and New Directions},
  author={Wei et al. (2019)},
  year={2019},
  note={arXiv:1908.01883}
}
```

- arXiv: 1908.01883