Dfm Dialogue Eval

---
name: dfm-dialogue-eval
description: Evaluates a unified dialogue foundation model across representation, knowledge distillation, and generation capabilities on diverse dialogue-oriented tasks. It probes the model's ability to perform intent detection, slot filling, semantic parsing, dialogue state tracking, text-to-SQL, and end-to-end task-oriented dialogue generation. Use when the user wants to benchmark on DialoGLUE, MULTIWOZ2.0, MULTIWOZ2.2, Spider, CoSQL, CLINC150, BANKING77, HWU64, RESTAURANT8K, DSTC8, TOP, PERSONALCHAT, COQA, SAMSUM, CANARD, or asks about evaluating this task. Reports exact match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.12662
  bibtex_key: chen2022dfm
  confidence: high
---

# dfm-dialogue-eval

> DFM: Dialogue Foundation Model for Universal Large-Scale Dialogue-Oriented Task Learning — Zhi Chen et al. (arXiv:2205.12662, 2022)

## What this evaluates

Evaluates a unified dialogue foundation model across representation, knowledge distillation, and generation capabilities on diverse dialogue-oriented tasks. It probes the model's ability to perform intent detection, slot filling, semantic parsing, dialogue state tracking, text-to-SQL, and end-to-end task-oriented dialogue generation.

## Datasets

- **DialoGLUE** — total ?; splits: test (-1), dev (-1)
- **MULTIWOZ2.0** — total ?; splits: test (-1)
- **MULTIWOZ2.2** — total ?; splits: test (-1)
- **Spider** — total ?; splits: dev (-1)
- **CoSQL** — total ?; splits: dev (-1)
- **CLINC150** — total ?; splits: test (-1)
- **BANKING77** — total ?; splits: test (-1)
- **HWU64** — total ?; splits: test (-1)
- **RESTAURANT8K** — total ?; splits: test (-1)
- **DSTC8** — total ?; splits: test (-1)
- **TOP** — total ?; splits: test (-1)
- **PERSONALCHAT** — total ?; splits: test (-1)
- **COQA** — total ?; splits: test (-1)
- **SAMSUM** — total ?; splits: test (-1)
- **CANARD** — total ?; splits: test (-1)

## Metrics

- `exact match (EM)` **(primary)** — range: [0, 1]
  - Percentage of predictions that exactly match the gold structured logical form or SQL query.
- `accuracy (ACC.)` — range: [0, 1]
  - Percentage of correctly predicted intents or labels.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for slot filling predictions.
- `joint goal accuracy (JGA)` — range: [0, 1]
  - Percentage of dialogue turns where all slot values are predicted correctly.
- `BLEU-4` — range: [0, 1]
  - 4-gram precision score measuring n-gram overlap between generated response and reference.
- `Combined` — range: [0, 1]
  - 0.5*(Inform+Success)+BLEU, where Inform and Success measure task completion constraints and entity matching.

## Input / output format

**Input**: Dialogue context or task instruction formatted as text-to-text prompts.

**Output**: Structured logical forms (for distillation tasks) or natural language responses (for generation tasks).

## Scoring recipe

```python
def score(predictions, gold, task):
    if task in ['intent', 'slot', 'qa', 'text2sql']:
        return exact_match_or_f1(predictions, gold)
    elif task == 'dst':
        return joint_goal_accuracy(predictions, gold)
    elif task in ['generation', 'tod']:
        return bleu4(predictions, gold)
    elif task == 'tod_combined':
        inform = check_constraints(predictions, gold)
        success = check_entities(predictions, gold)
        return 0.5 * (inform + success) + bleu4(predictions, gold)
```

## Common pitfalls

- Mixing up zero-shot (unseen in training) vs fine-tuned settings; DSTC8 and TOP are explicitly noted as unseen/zero-shot.
- Using different decoding strategies (Beam vs PICARD) significantly changes text-to-SQL scores and must be reported consistently.
- DialoGLUE requires preserving specific downstream architectures (e.g., Trippy for DST) during fine-tuning; altering them invalidates comparisons.

## Evidence (verbatim from paper)

> BLEU is BLEU-4 score, which measures the response consistency. Combined score equals to 0.5*(Inform+Success)+BLEU.

## Citation

```bibtex
@misc{chen2022dfm,
  title={DFM: Dialogue Foundation Model for Universal Large-Scale Dialogue-Oriented Task Learning},
  author={Zhi Chen et al.},
  year={2022},
  note={arXiv:2205.12662}
}
```

- arXiv: 2205.12662