Dph Alignment Eval

---
name: dph-alignment-eval
description: Evaluates language models on natural language understanding, commonsense reasoning, and reading comprehension to measure alignment quality and reasoning preservation. It compares standard log-probability predictions against scores derived from a learned Direct Preference Head (DPH) reward model to assess self-evaluation capabilities. Use when the user wants to benchmark on GLUE, RACE, ARC, OpenBookQA, HellaSwag, WinoGrande, BoolQ, PIQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.20053
  bibtex_key: asadahadji2024directpreferenceheads
  confidence: high
---

# dph-alignment-eval

> Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads — Asada Hadji-Kyriacou et al. (2024) (arXiv:2405.20053, 2024)

## What this evaluates

Evaluates language models on natural language understanding, commonsense reasoning, and reading comprehension to measure alignment quality and reasoning preservation. It compares standard log-probability predictions against scores derived from a learned Direct Preference Head (DPH) reward model to assess self-evaluation capabilities.

## Datasets

- **GLUE** — total ?; splits: test (-1)
- **RACE** — total ?; splits: test (-1)
- **ARC** — total ?; splits: test (-1)
- **OpenBookQA** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: val (-1)
- **WinoGrande** — total ?; splits: val (-1)
- **BoolQ** — total ?; splits: val (-1)
- **PIQA** — total ?; splits: val (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted options or answers out of total instances. Calculated per task and averaged for suite-level scores.
- `GLUE average score` — range: [0, 1]
  - Macro-average of task-specific scores (accuracy, F1, or correlation) across GLUE subtasks, reported with and without WNLI.
- `Spearman/Pearson correlation` — range: [-1, 1]
  - Rank (Spearman) or linear (Pearson) correlation coefficient between predicted and gold scores for regression tasks like CoLA and STS-B.

## Input / output format

**Input**: Text prompts containing a question or context followed by multiple-choice options or a sentence completion target.

**Output**: Model-generated text or selected option index. Predictions are obtained either from highest log-probability completions or by selecting the option with the highest DPH reward score.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)

def compute_glue_avg(task_scores):
    return sum(task_scores.values()) / len(task_scores)

def compute_correlation(preds, golds):
    # Use scipy.stats.spearmanr or pearsonr
    return correlation_coefficient(preds, golds)
```

## Common pitfalls

- GLUE average score is reported both with and without WNLI; excluding WNLI changes the macro-average significantly.
- Different datasets use different splits (test vs. validation) as specified in the methodology, which must be respected to match reported scores.
- DPH predictions are derived from reward scores on hidden states, not directly from the language modeling head's log-probabilities; confusing the two leads to incorrect baseline comparisons.

## Evidence (verbatim from paper)

> With SFT alone we are able to attain comparable performance to TinyLlama using half as many parameters, and when applying DPH alignment we achieve a 7.2% increase over the TinyLlama average score and the highest accuracy in 5 of the 7 tasks.

## Citation

```bibtex
@misc{asadahadji2024directpreferenceheads,
  title={Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads},
  author={Asada Hadji-Kyriacou et al. (2024)},
  year={2024},
  note={arXiv:2405.20053}
}
```

- arXiv: 2405.20053