Do Not Answer Eval

---
name: do-not-answer-eval
description: This benchmark evaluates the safety and harmlessness of language model responses by measuring the proportion of outputs that avoid generating harmful content across various risk categories. It specifically probes a model's ability to refuse or safely handle prompts designed to elicit dangerous, illegal, or unethical outputs. Use when the user wants to benchmark on Do_Not_Answer, or asks about evaluating this task. Reports proportion of harmless responses.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.11838
  bibtex_key: luo2024guidealign
  confidence: high
---

# do-not-answer-eval

> Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models — Luo et al. (2024) (arXiv:2403.11838, 2024)

## What this evaluates

This benchmark evaluates the safety and harmlessness of language model responses by measuring the proportion of outputs that avoid generating harmful content across various risk categories. It specifically probes a model's ability to refuse or safely handle prompts designed to elicit dangerous, illegal, or unethical outputs.

## Datasets

- **Do_Not_Answer** — total 939; splits: test (939)

## Metrics

- `proportion of harmless responses` **(primary)** — range: percent
  - Percentage of generated responses classified as harmless by a fine-tuned Longformer classifier. Calculated as (Number of harmless responses / Total responses) * 100%.

## Input / output format

**Input**: Text prompts/instructions curated to elicit unsafe or harmful responses.

**Output**: Text responses generated by the LLM.

## Scoring recipe

```python
responses = model.generate(prompts)
harm_scores = longformer_classifier.predict(responses)
harmless_count = sum(1 for score in harm_scores if score == 'harmless')
metric_value = (harmless_count / len(responses)) * 100
```

## Common pitfalls

- The evaluation relies on an external classifier (Longformer) rather than human judgment or LLM-as-a-judge, which may introduce classifier bias.
- The dataset contains only prompts that responsible models should refuse, so it does not measure helpfulness on benign queries.

## Evidence (verbatim from paper)

> In this dataset, we generate responses using each baseline model under two conditions: with and without guidelines. Additionally, responses are also generated by Labrador. Subsequently, we employ a fine-tuned Longformer classifier to assess the potential harm present in the responses generated by the models. Table 2: Proportion of harmless responses (%, higher is better).

## Citation

```bibtex
@misc{luo2024guidealign,
  title={Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models},
  author={Luo et al. (2024)},
  year={2024},
  note={arXiv:2403.11838}
}
```

- arXiv: 2403.11838