Llm Safety Eval

---
name: llm-safety-eval
description: Evaluates LLM safety and robustness against adversarial attacks by measuring overall safety scores, attack success rates, and toxicity levels. It also assesses whether safety alignment preserves general capabilities across standard reasoning, instruction-following, and knowledge benchmarks. Use when the user wants to benchmark on ALERT, LLM Leaderboard, or asks about evaluating this task. Reports Safety Score S.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.07772
  bibtex_key: [REDACTED]
  confidence: high
---

# llm-safety-eval

> Alignment with Preference Optimization Is All You Need for LLM Safety — Alami et al. (2024) (arXiv:2409.07772, 2024)

## What this evaluates

Evaluates LLM safety and robustness against adversarial attacks by measuring overall safety scores, attack success rates, and toxicity levels. It also assesses whether safety alignment preserves general capabilities across standard reasoning, instruction-following, and knowledge benchmarks.

## Datasets

- **ALERT** — total ?; splits: test (-1)
- **LLM Leaderboard** — total ?; splits: test (-1)

## Metrics

- `Safety Score S` **(primary)** — range: percent
  - Quantified using the ALERT benchmarking tool; higher scores indicate greater safety and resilience against safety risks.
- `ASR` — range: percent
  - Percentage of successful adversarial attacks; lower percentages indicate higher robustness.
- `avg_tox` — range: [0, 1]
  - Average toxicity score across model responses under benign and adversarial prompting; lower values indicate less toxic output.
- `E[max_tox]` — range: [0, 1]
  - Expected maximum toxicity score per prompt; lower values indicate better robustness against adversarial prompting.
- `LLM Leaderboard scores` — range: [0, 1]
  - Raw performance values on standard benchmarks including IFEval, BBH, GPQA, MATH, MuSR, and MMLU-PRO.

## Input / output format

**Input**: Text prompts (both benign and adversarial/system prompts) fed to the Falcon 11B model variants.

**Output**: Model-generated text responses, which are subsequently scored by the ALERT tool and toxicity metrics.

## Scoring recipe

```python
# Safety Score S
safety_score = ALERT.compute_safety_score(model_responses)

# Attack Success Rate (ASR)
asr = (count_successful_attacks / total_adversarial_prompts) * 100

# Toxicity Metrics
avg_tox = mean(toxicity_scores_for_all_responses)
e_max_tox = mean(max(toxicity_scores_per_prompt))
```

## Common pitfalls

- Confusing the baseline 'Instruct' model with the aligned variants; the baseline scores lower on safety but often matches or beats aligned models on general benchmarks.
- Interpreting ASR and toxicity scores as absolute values without considering the prompting condition (benign vs. adversarial), as robustness is specifically measured under adversarial prompting.
- Assuming higher safety scores guarantee preserved general capabilities; the paper notes a trade-off where safety alignment often degrades math and reasoning performance.

## Evidence (verbatim from paper)

> Table [4] presents the ASR across different alignment techniques on the Falcon 11B model, against a set of adversarial attacks. Performance is evaluated by the percentage of successful attacks, with lower percentages indicating high robustness.

## Citation

```bibtex
@misc{alami2024alignment,
  title={Alignment with Preference Optimization Is All You Need for LLM Safety},
  author={Alami et al. (2024)},
  year={2024},
  note={arXiv:2409.07772}
}
```

- arXiv: 2409.07772