Persona Chat Eval

---
name: persona-chat-eval
description: Evaluates a dialogue agent's ability to generate or rank contextually appropriate next utterances while maintaining consistency with a given personal profile. It probes the model's capacity for persona-conditioned chit-chat and its ability to infer or reflect speaker interests during conversation. Use when the user wants to benchmark on PersonaChat, or asks about evaluating this task. Reports Hits@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1801.07243
  bibtex_key: zhang2018personalizing
  confidence: high
---

# persona-chat-eval

> Personalizing Dialogue Agents: I have a dog, do you have pets too? — Zhang et al. (2018) (arXiv:1801.07243, 2018)

## What this evaluates

Evaluates a dialogue agent's ability to generate or rank contextually appropriate next utterances while maintaining consistency with a given personal profile. It probes the model's capacity for persona-conditioned chit-chat and its ability to infer or reflect speaker interests during conversation.

## Datasets

- **PersonaChat** — total 162064; splits: train (-1), test (-1)

## Metrics

- `Hits@1` **(primary)** — range: [0, 1]
  - Fraction of test instances where the correct next utterance appears as the top-ranked candidate among a set of options. For generative models, it measures whether the generated response exactly matches the ground truth or a candidate in the ranking pool.
- `Perplexity` — range: [0, inf)
  - Exponential of the average negative log-likelihood of the ground truth next utterance under the model's probability distribution. Lower values indicate better predictive performance.
- `Human Ratings (Fluency, Engagingness, Consistency)` — range: [1, 5]
  - Crowdsourced workers rate model responses on a 1-5 Likert scale across three dimensions after conversing with the agent. Higher scores indicate better perceived dialogue quality.
- `Persona Detection Accuracy` — range: [0, 1]
  - Binary accuracy measuring whether a human conversational partner correctly identifies the model's assigned persona from two candidate profiles after a dialogue.

## Input / output format

**Input**: Dialogue history concatenated with the persona profile(s) of the speaker(s) to be conditioned on.

**Output**: For generative models: a single natural language next utterance. For ranking models: a ranked list of candidate responses.

## Scoring recipe

```python
def compute_hits_at_1(predictions, candidates, ground_truth):
    correct = 0
    for pred, gt in zip(predictions, ground_truth):
        if pred == gt or gt in candidates:
            correct += 1
    return correct / len(predictions)

# Human metrics are averaged 1-5 Likert scores across raters for fluency, engagingness, and consistency.
# Persona detection accuracy is the fraction of correct binary choices.
```

## Common pitfalls

- Automated metrics like perplexity and hits@1 are notoriously poor proxies for actual dialogue quality and engagement.
- Word-overlap-based evaluation can be inflated by trivial responses that match persona keywords without semantic coherence.
- Human evaluation scores exhibit high variance, making subtle model differences hard to detect without large sample sizes.

## Evidence (verbatim from paper)

> Overall, the results show the following key points: Persona Conditioning Most models improve significantly when conditioning prediction on their own persona at least for the original (non-revised) versions, which is an easier task than the revised ones which have no word overlap. For example, the Profile Memory generation model has improved perplexity and hits@1 compared to Seq2Seq, and all the ranking algorithms (IR baseline, Starspace and Profile Memory Networks) obtain improved hits@1.

## Citation

```bibtex
@misc{zhang2018personalizing,
  title={Personalizing Dialogue Agents: I have a dog, do you have pets too?},
  author={Zhang et al. (2018)},
  year={2018},
  note={arXiv:1801.07243}
}
```

- arXiv: 1801.07243