Constitution Ai Feedback Eval

---
name: constitution-ai-feedback-eval
description: This evaluation probes how different instructional guidelines (constitutions) shape AI-generated medical dialogues across specific socio-communicative dimensions like empathy, information gathering, and decision-making. It measures human preference for dialogue quality under varying constitutional constraints. Use when the user wants to benchmark on Custom AI-generated medical dialogues, or asks about evaluating this task. Reports Bradley-Terry preference rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.10168
  bibtex_key: redgate2024constitutions
  confidence: high
---

# constitution-ai-feedback-eval

> Evaluating the role of `Constitutions' for learning from AI feedback — Redgate et al. (2024) (arXiv:2411.10168, 2024)

## What this evaluates

This evaluation probes how different instructional guidelines (constitutions) shape AI-generated medical dialogues across specific socio-communicative dimensions like empathy, information gathering, and decision-making. It measures human preference for dialogue quality under varying constitutional constraints.

## Datasets

- **Custom AI-generated medical dialogues** — total ?; splits: test (-1)

## Metrics

- `Bradley-Terry preference rate` **(primary)** — range: [0, 1]
  - Estimated from pairwise human preferences using a Bradley-Terry model. Represents the relative likelihood that a dialogue generated under a specific constitution is preferred over others for a given evaluation dimension.

## Input / output format

**Input**: AI-generated medical dialogue transcripts conditioned on different constitutional guidelines.

**Output**: Human pairwise preference ratings across six communication dimensions (Fostering Relationship, Gathering Information, Providing Information, Decision Making, Responding to Emotions, Enabling treatment behaviour).

## Scoring recipe

```python
# predictions: list of human pairwise choices (winner_constitution, loser_constitution) per dimension
# gold: constitution labels
counts = defaultdict(int)
for winner, loser in predictions:
    counts[winner] += 1
    counts[loser] += 1
# Fit Bradley-Terry model to estimate log-odds preference parameters
bt_params = fit_bradley_terry(counts)
# Convert to preference rate for reporting
return {dim: sigmoid(bt_params[dim]) for dim in dimensions}
```

## Common pitfalls

- Error bars represent 95% confidence intervals not adjusted for multiple comparisons.
- Holistic constitution selection showed no clear pattern despite significant dimension-specific preferences.
- Participants often disliked verbose or overly emotive responses even when rating them as more empathetic.

## Evidence (verbatim from paper)

> In Figure 2, we show the rate at which the conversations generated according to each constitution are preferred to the others for each dimension of evaluation, alongside the estimated parameters for a Bradley-Terry model.

## Citation

```bibtex
@misc{redgate2024constitutions,
  title={Evaluating the role of `Constitutions' for learning from AI feedback},
  author={Redgate et al. (2024)},
  year={2024},
  note={arXiv:2411.10168}
}
```

- arXiv: 2411.10168