Hhh Alignment Eval

---
name: hhh-alignment-eval
description: This benchmark evaluates language model alignment across four dimensions: helpfulness, honesty, harmlessness, and other. It measures how well a model's outputs adhere to human values and safety guidelines through pairwise comparison. Use when the user wants to benchmark on HHH_Alignment, or asks about evaluating this task. Reports Net Win Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.11838
  bibtex_key: luo2024guidealign
  confidence: high
---

# hhh-alignment-eval

> Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models — Luo et al. (2024) (arXiv:2403.11838, 2024)

## What this evaluates

This benchmark evaluates language model alignment across four dimensions: helpfulness, honesty, harmlessness, and other. It measures how well a model's outputs adhere to human values and safety guidelines through pairwise comparison.

## Datasets

- **HHH_Alignment** — total 103; splits: test (103)

## Metrics

- `Net Win Rate` **(primary)** — range: [-1, 1]
  - Calculated as (Win - Lose) / (Win + Lose), where Win, Tie, and Lose represent the outcomes of pairwise response comparisons judged by GPT-4. Expressed as a percentage or ratio.

## Input / output format

**Input**: Prompts categorized into helpfulness, honesty, harmlessness, and other.

**Output**: Text responses generated by the LLM.

## Scoring recipe

```python
pairs = generate_pairwise_prompts(model_a, model_b)
judgments = gpt4_judge.evaluate(pairs)
wins = count(judgments == 'A wins')
loses = count(judgments == 'B wins')
net_win_rate = (wins - loses) / (wins + loses)
```

## Common pitfalls

- GPT-4 is used as the judge, which may introduce position bias or preference for its own style; the paper notes reversing option order to mitigate this.
- The benchmark mixes capability and safety evaluation, making it hard to isolate pure safety performance.

## Evidence (verbatim from paper)

> We task GPT-4 with selecting the superior response from a pair of responses. To mitigate potential biases introduced by the order of options, we systematically reversed the option order, facilitating two evaluations for each question. Table 3 illustrates all four experimental sets’ outcomes. The value of "Net Win Rate" is calculated by dividing the difference between the value of "Win" and "Lose" for each row by the sum of the values for that row.

## Citation

```bibtex
@misc{luo2024guidealign,
  title={Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models},
  author={Luo et al. (2024)},
  year={2024},
  note={arXiv:2403.11838}
}
```

- arXiv: 2403.11838