Tod Eval

---
name: tod-eval
description: Evaluates a model's ability to perform multi-turn task-oriented dialogue by jointly tracking dialogue states and generating task-completing responses. It probes how well the system understands user intents, fills correct slots across multiple domains, and fulfills explicit user requests. Use when the user wants to benchmark on MultiWOZ2.0/2.1, In-Car, or asks about evaluating this task. Reports Comb.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.14457
  bibtex_key: du2024rewarding
  confidence: high
---

# tod-eval

> Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue — Du et al. (2024) (arXiv:2406.14457, 2024)

## What this evaluates

Evaluates a model's ability to perform multi-turn task-oriented dialogue by jointly tracking dialogue states and generating task-completing responses. It probes how well the system understands user intents, fills correct slots across multiple domains, and fulfills explicit user requests.

## Datasets

- **MultiWOZ2.0/2.1** — total 10438; splits: train (8438), val (1000), test (1000); HF `multiwoz`
- **In-Car** — total 3031; splits: train (2425), val (302), test (304)

## Metrics

- `Inform` — range: [0, 1]
  - Measures whether the system provides correct entities based on its understanding of user requirements.
- `Success` — range: [0, 1]
  - Measures whether all user requests are met by the system.
- `BLEU` — range: [0, 100]
  - Standard n-gram overlap metric used to measure the fluency of the generated response.
- `Comb` **(primary)** — range: [0, 100]
  - Overall quality measure computed as (Inform + Success) × 0.5 + BLEU.
- `Match` — range: [0, 1]
  - Measures if a system can track all correct dialogue states to satisfy the user.
- `SuccF1` — range: [0, 1]
  - Improves on Success by considering both recall (completeness) and precision (accuracy) to gauge response accuracy and completeness.

## Input / output format

**Input**: Multi-turn dialogue history with delexicalized system responses and user utterances, including domain constraints and slot values.

**Output**: Delexicalized system response and/or predicted dialogue state (slot values).

## Scoring recipe

```python
def score_multiwoz(pred, gold):
    inform = 1.0 if set(pred.entities) == set(gold.entities) else 0.0
    success = 1.0 if all(req in pred.response for req in gold.requests) else 0.0
    bleu = compute_bleu(gold.response, pred.response)
    comb = (inform + success) * 0.5 + bleu
    return {'Inform': inform, 'Success': success, 'BLEU': bleu, 'Comb': comb}

def score_in_car(pred, gold):
    match = 1.0 if pred.states == gold.states else 0.0
    succ_f1 = f1_score(gold.requests, pred.requests)
    return {'Match': match, 'SuccF1': succ_f1}
```

## Common pitfalls

- BLEU scores are often low because the model optimizes for task completion and understanding rather than linguistic fluency.
- Delexicalized responses require exact entity matching; mismatched entity IDs can artificially penalize Inform/Success scores.
- The Comb score masks trade-offs between understanding (Inform) and generation (BLEU), so reporting individual components is necessary.

## Evidence (verbatim from paper)

> For MultiWOZ, we report Inform and Success as introduced in [Section 4.1]. Additionally, we report BLEU that is used to measure the fluency of the generated response. Consequently, we report (Comb) that is computed by (Inform + Success) ×0.5 + BLEU as an overall quality measure. For In-Car, we leverage Match to measure if a system can track all correct states to satisfy the user. SuccF1 improves on the Success by considering both how completely (recall) and accurately (precision) the system handles requests.

## Citation

```bibtex
@misc{du2024rewarding,
  title={Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue},
  author={Du et al. (2024)},
  year={2024},
  note={arXiv:2406.14457}
}
```

- arXiv: 2406.14457