Cmu Doq Eval

---
name: cmu-doq-eval
description: Evaluates the quality of generated responses in document-grounded conversations, specifically measuring how well models leverage external document context to produce engaging and fluent multi-turn dialogue. It assesses both automatic language modeling metrics and human-perceived response quality. Use when the user wants to benchmark on CMU.DoG, or asks about evaluating this task. Reports Perplexity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1809.07358
  bibtex_key: zhou2018dataset
  confidence: high
---

# cmu-doq-eval

> A Dataset for Document Grounded Conversations — Zhou et al. (2018) (arXiv:1809.07358, 2018)

## What this evaluates

Evaluates the quality of generated responses in document-grounded conversations, specifically measuring how well models leverage external document context to produce engaging and fluent multi-turn dialogue. It assesses both automatic language modeling metrics and human-perceived response quality.

## Datasets

- **CMU.DoG** — total 4112; splits: train (-1), test (-1); repo https://github.com/festvox/datasets-CMU_DoG

## Metrics

- `Perplexity` **(primary)** — range: other
  - Standard n-gram language model perplexity computed on generated test responses. Lower values indicate better fluency and language modeling performance.
- `Engagement Preference Rate` — range: percent
  - Percentage of times a model's response is chosen over the baseline in a pairwise comparison given a 1-utterance chat history, with majority vote from 3 annotators.
- `Fluency Score` — range: [1, 4]
  - Mean rating on a 4-point Likert scale (1=unreadable, 4=perfectly readable) assigned by 3 annotators per response.

## Input / output format

**Input**: Chat history (1 utterance) and corresponding document section/context for model generation; for human evaluation, chat history is presented to annotators alongside model responses.

**Output**: Generated response/utterance for the model; for human evaluation, a preference label (SEQ, SEQS, or No Preference) or a fluency score (1-4).

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    # Perplexity
    ppl = exp(-mean(log_prob(predictions, ngram_model)))
    
    # Engagement Preference Rate
    votes = [pairwise_compare(history, pred, gold) for pred, gold in zip(predictions, golds)]
    engagement_rate = sum(votes == 'preferred') / len(predictions)
    
    # Fluency Score
    fluency_scores = [fluency_rating(pred) for pred in predictions]
    fluency_score = mean(fluency_scores)
    
    return ppl, engagement_rate, fluency_score
```

## Common pitfalls

- The engagement metric includes a 'No Preference' option, so win rates for individual models do not sum to 100%.
- Fluency scores are averaged across 3 unique annotators per response, not calculated from raw individual ratings.
- Perplexity is computed using a custom n-gram model trained specifically on the training set responses, not a standard pre-trained language model.

## Evidence (verbatim from paper)

> To automatically evaluate the fluency of the models, we use perplexity measure. We build a language model on the train set of responses using ngrams up to an order of $3^{3}$. The generated test responses achieve a perplexity of 21.8 for the SEQ model and 10.11 for the SEQS model. We also perform two kinds of human evaluations to evaluate the quality of predicted utterances - engagement and fluency. The workers were asked to evaluate the fluency of the generated response on a scale of 1 to 4, where 1 is unreadable and 4 is perfectly readable.

## Citation

```bibtex
@misc{zhou2018dataset,
  title={A Dataset for Document Grounded Conversations},
  author={Zhou et al. (2018)},
  year={2018},
  note={arXiv:1809.07358}
}
```

- arXiv: 1809.07358