Dialogstudio Response Eval

---
name: dialogstudio-response-eval
description: Evaluates a model's ability to generate task-oriented and knowledge-grounded dialogue responses in zero-shot and few-shot settings. It measures lexical overlap and unigram F1 against ground-truth responses to assess generalization across open-domain and multi-domain conversational tasks. Use when the user wants to benchmark on CoQA, MultiWOZ 2.2, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.10172
  bibtex_key: zhang2023dialogstudio
  confidence: high
---

# dialogstudio-response-eval

> DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI — Jianguo Zhang et al. (2023) (arXiv:2307.10172, 2023)

## What this evaluates

Evaluates a model's ability to generate task-oriented and knowledge-grounded dialogue responses in zero-shot and few-shot settings. It measures lexical overlap and unigram F1 against ground-truth responses to assess generalization across open-domain and multi-domain conversational tasks.

## Datasets

- **CoQA** — total ?; splits: test (-1)
- **MultiWOZ 2.2** — total ?; splits: test (-1)

## Metrics

- `ROUGE-L` **(primary)** — range: percent
  - Measures the longest common subsequence (LCS) overlap between the generated response and the ground-truth response.
- `F1` — range: percent
  - Unigram F1 overlap score between the prediction and ground-truth response, as defined by Miller et al. (2017).

## Input / output format

**Input**: Instruction prompt containing dialogue history formatted with <USER> and <SYSTEM> tokens, optionally including <EXTERNAL KNOWLEDGE> or dialogue acts, followed by the user's latest utterance.

**Output**: A single text string representing the system's next dialogue response.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    rouge_l_scores = [rouge_l_score(pred, ref) for pred, ref in zip(predictions, references)]
    f1_scores = [unigram_f1_score(pred, ref) for pred, ref in zip(predictions, references)]
    return {
        "ROUGE-L": sum(rouge_l_scores) / len(rouge_l_scores),
        "F1": sum(f1_scores) / len(f1_scores)
    }
```

## Common pitfalls

- Data leakage: CoQA and MultiWOZ 2.2 must be strictly excluded from the pre-training dataset to ensure valid zero-shot evaluation.
- Prompt formatting: Models must follow the exact instruction template (e.g., 'Continue the dialogue as a task-oriented dialogue system called SYSTEM...') to match the evaluation setup.

## Evidence (verbatim from paper)

> We focus on zero-shot evaluation and report the ROUGE-L and F1 score (Miller et al., 2017), where ROUGE-L measures the longest common subsequence and F1 measures the Unigram F1 overlap between the prediction and ground-truth response.

## Citation

```bibtex
@misc{zhang2023dialogstudio,
  title={DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI},
  author={Jianguo Zhang et al. (2023)},
  year={2023},
  note={arXiv:2307.10172}
}
```

- arXiv: 2307.10172