Gcai Constitution Eval

---
name: gcai-constitution-eval
description: Evaluates the moral grounding, coherence, fairness, and real-world applicability of AI alignment constitutions through human surveys, alongside the downstream safety alignment and general capabilities of fine-tuned language models. Use when the user wants to benchmark on BABELSCAPE/ALERT, MMLU, Social Bias BBQ, or asks about evaluating this task. Reports 5-point Likert rating.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.18760
  bibtex_key: bell2026beyondpreferences
  confidence: high
---

# gcai-constitution-eval

> Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values — Bell et al. (2026) (arXiv:2601.18760, 2026)

## What this evaluates

Evaluates the moral grounding, coherence, fairness, and real-world applicability of AI alignment constitutions through human surveys, alongside the downstream safety alignment and general capabilities of fine-tuned language models.

## Datasets

- **BABELSCAPE/ALERT** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **Social Bias BBQ** — total ?; splits: test (-1)

## Metrics

- `5-point Likert rating` **(primary)** — range: [1, 5]
  - Participants rate each principle or constitution on a 5-point Likert scale indicating agreement or quality along specific dimensions such as Moral Grounding, Fairness, and Coherence.
- `Pairwise preference selection` — range: categorical
  - Participants choose which of two constitutions they prefer personally and for governing AI behavior for the U.S. population.
- `MMLU` — range: [0, 1]
  - Standard multiple-choice accuracy across 57 subjects measuring general multi-domain language understanding.
- `Social Bias BBQ` — range: [0, 1]
  - Measures stereotyping under ambiguous and disambiguated social contexts to evaluate social bias in aligned models.

## Input / output format

**Input**: Constitution pairs or individual principles for human surveys; red-teaming prompts, MMLU multiple-choice questions, and BBQ social context questions for model evaluation.

**Output**: Human surveys: binary preference choice, free-text justification, and 5-point Likert ratings per dimension. Models: generated text responses or selected multiple-choice answers.

## Scoring recipe

```python
# Human Survey Scoring
likert_scores = [r for r in ratings if 1 <= r <= 5]
dimension_avg = sum(likert_scores) / len(likert_scores)
preference_win = 1 if chosen == target else 0

# Benchmark Scoring
mmlu_acc = sum(correct_preds) / total_questions
bbq_bias = sum(stereotypical_preds) / total_questions
```

## Common pitfalls

- Human raters may conflate personal moral beliefs with societal applicability, skewing preference scores.
- Qualitative coding of red-teaming responses relies on researcher double-coding, introducing potential subjectivity.
- Abstract dimensions like Constancy and Faithfulness lack standardized rubrics, limiting cross-study comparability.

## Evidence (verbatim from paper)

> We benchmark on MMLU [21] to measure general multi-domain language understanding, and Social Bias BBQ [32] to test whether models exhibit stereotyping under ambiguous and disambiguated social contexts (See §7.1).

## Citation

```bibtex
@misc{bell2026beyondpreferences,
  title={Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values},
  author={Bell et al. (2026)},
  year={2026},
  note={arXiv:2601.18760}
}
```

- arXiv: 2601.18760