Convai2 Eval

---
name: convai2-eval
description: Evaluates open-domain chatbot capabilities in persona-driven multi-turn conversations, measuring response quality via automatic metrics and human judgments of engagement and persona consistency. Use when the user wants to benchmark on PERSONA-CHAT, or asks about evaluating this task. Reports Engagingness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1902.00098
  bibtex_key: dinan2019convai2
  confidence: high
---

# convai2-eval

> The Second Conversational Intelligence Challenge (ConvAI2) — Dinan et al. (2019) (arXiv:1902.00098, 2019)

## What this evaluates

Evaluates open-domain chatbot capabilities in persona-driven multi-turn conversations, measuring response quality via automatic metrics and human judgments of engagement and persona consistency.

## Datasets

- **PERSONA-CHAT** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/DeepPavlov/convai

## Metrics

- `F1` — range: [0, 1]
  - Word overlap F1 score between the generated response and the ground truth response.
- `Hits@1` — range: [0, 1]
  - Fraction of instances where the model ranks the correct response as the top candidate among a set of candidates.
- `Perplexity` — range: other
  - Exponential of the average negative log-likelihood of the ground truth response tokens under the model's language model distribution.
- `Engagingness` **(primary)** — range: [1, 4]
  - Human annotator rating on a 1-4 scale answering 'How much did you enjoy talking to this user?' after a 4-6 turn dialogue.
- `Persona Detection` — range: [0, 1]
  - Fraction of times the human correctly identifies the model's assigned persona versus a random distractor persona.

## Input / output format

**Input**: Multi-turn conversation history with assigned personas for both human and model, followed by a prompt for the model to generate the next utterance.

**Output**: A single text response string.

## Scoring recipe

```python
def score(predictions, golds, candidates=None, human_ratings=None):
    # Automatic metrics
    f1_vals = [f1_score(g.split(), p.split()) for p, g in zip(predictions, golds)]
    hits1 = sum(1 for c, g in zip(candidates, golds) if c[0] == g) / len(golds)
    ppl = exp(-mean([log_prob(g) for g in golds]))
    # Human metrics
    engagingness = mean(human_ratings) if human_ratings else None
    persona_det = mean([1 if correct else 0 for correct in human_ratings]) if human_ratings else None
    return {'F1': mean(f1_vals), 'Hits@1': hits1, 'Perplexity': ppl, 'Engagingness': engagingness, 'Persona Detection': persona_det}
```

## Common pitfalls

- F1 score is easily gamed by outputting frequent training words without coherent dialogue, as shown by a toy baseline outperforming all models.
- Automatic metrics (F1, Hits@1) correlate poorly with human engagement judgments, causing leaderboard rankings to misrepresent actual chatbot quality.
- Unpaid 'wild' evaluations yield noisy, off-instruction dialogues that lack systematic filtering criteria and were ultimately discarded.

## Evidence (verbatim from paper)

> The rank of each team was determined by sorting by the minimum rank of the score in any of the three metrics (F1, Hits@1, and Perplexity). ... For each evaluation, we paired a human worker with a model, assigned each of them personas, and instructed the humans to chat with and get to know their partner. Dialogues were of length 4-6 turns each. Following a short conversation, we asked workers "How much did you enjoy talking to this user?" and had them answer on a scale of 1-4.

## Citation

```bibtex
@misc{dinan2019convai2,
  title={The Second Conversational Intelligence Challenge (ConvAI2)},
  author={Dinan et al. (2019)},
  year={2019},
  note={arXiv:1902.00098}
}
```

- arXiv: 1902.00098