Fortress Eval

---
name: fortress-eval
description: Evaluates LLM safeguard robustness against national security and public safety (NSPS) risks by measuring both the model's tendency to generate harmful content in response to adversarial prompts and its tendency to incorrectly refuse legitimate benign requests. Use when the user wants to benchmark on FORTRESS, or asks about evaluating this task. Reports Average Risk Score (ARS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.14922
  bibtex_key: knight2025fortress
  confidence: high
---

# fortress-eval

> FORTRESS: Frontier Risk Evaluation for National Security and Public Safety — Knight et al. (2025) (arXiv:2506.14922, 2025)

## What this evaluates

Evaluates LLM safeguard robustness against national security and public safety (NSPS) risks by measuring both the model's tendency to generate harmful content in response to adversarial prompts and its tendency to incorrectly refuse legitimate benign requests.

## Datasets

- **FORTRESS** — total 1000; splits: test (1000); HF `ScaleAI/fortress_public`

## Metrics

- `Average Risk Score (ARS)` **(primary)** — range: percent
  - Percentage of adversarial prompts that successfully elicit harmful content, averaged across three categories (CBRNE, political violence, illicit financial activities). Computed using 4–7 binary rubrics per instance.
- `Over-Refusal Score (ORS)` — range: percent
  - Percentage of benign prompts that are incorrectly refused by the model, averaged across the same three categories. Lower values indicate better user experience and fewer false positives.

## Input / output format

**Input**: Adversarial or benign prompts targeting national security and public safety domains (CBRNE, political violence, illicit financial activities).

**Output**: Model-generated text response to the prompt.

## Scoring recipe

```python
def compute_ars(predictions, rubrics):
    violations = [1 if violates_rubric(pred, rubrics) else 0 for pred in predictions]
    return sum(violations) / len(violations) * 100

def compute_ors(predictions, is_benign):
    refusals = [1 if is_refusal(pred) else 0 for pred in predictions]
    return sum(refusals) / len(refusals) * 100
```

## Common pitfalls

- ARS and ORS are reported as percentages (0-100), not probabilities or raw counts.
- ORS specifically measures false refusals on benign inputs, not safety refusals on harmful ones; lower ORS is better.

## Evidence (verbatim from paper)

> The top section shows ARS, while the bottom section presents ORS that quantify each model's tendency to incorrectly reject legitimate requests. These models were not utilized in the benchmark's adversarial example collection phase, providing an independent assessment of their safety capabilities.

## Citation

```bibtex
@misc{knight2025fortress,
  title={FORTRESS: Frontier Risk Evaluation for National Security and Public Safety},
  author={Knight et al. (2025)},
  year={2025},
  note={arXiv:2506.14922}
}
```

- arXiv: 2506.14922