Prefeval Eval

---
name: prefeval-eval
description: Measures an agent's capability to retain and adhere to user preferences during long, multi-turn conversations. It tests whether the model can maintain consistency without external reminders or with explicit preference cues. Use when the user wants to benchmark on PrefEval, or asks about evaluating this task. Reports preference retention accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.03991
  bibtex_key: bao2025galaxy
  confidence: high
---

# prefeval-eval

> Galaxy: A Cognition-Centered Framework for Proactive, Privacy-Preserving, and Self-Evolving LLM Agents — Bao et al. (2025) (arXiv:2508.03991, 2025)

## What this evaluates

Measures an agent's capability to retain and adhere to user preferences during long, multi-turn conversations. It tests whether the model can maintain consistency without external reminders or with explicit preference cues.

## Datasets

- **PrefEval** — total ?; splits: test (-1)

## Metrics

- `preference retention accuracy` **(primary)** — range: percent
  - Accuracy of whether the agent correctly maintains user preferences across multi-round conversations, measured under Zero-Shot (no reminders) and Reminder conditions.

## Input / output format

**Input**: Long multi-turn conversational prompts containing explicit and implicit user preferences.

**Output**: Agent responses that must align with previously stated user preferences.

## Scoring recipe

```python
correct = 0
total = len(preference_checks)
for check in preference_checks:
  if agent_response_matches_preference(check):
    correct += 1
return (correct / total) * 100
```

## Common pitfalls

- Failing to distinguish between Zero-Shot and Reminder evaluation conditions.
- Evaluating preference retention on short conversations instead of long multi-turn contexts.

## Evidence (verbatim from paper)

> PrefEval focuses on whether agents can maintain user preferences in long conversations. It has two ways of measuring the accuracy of preference retention in multi-round conversations: without reminding users of their preferences (Zero-Shot) and by reminding users of their preferences (Reminder).

## Citation

```bibtex
@misc{bao2025galaxy,
  title={Galaxy: A Cognition-Centered Framework for Proactive, Privacy-Preserving, and Self-Evolving LLM Agents},
  author={Bao et al. (2025)},
  year={2025},
  note={arXiv:2508.03991}
}
```

- arXiv: 2508.03991