Statcan Dialogue Eval

---
name: statcan-dialogue-eval
description: Evaluates a model's ability to retrieve relevant statistical data tables from a large corpus based on conversational dialogue history, and its ability to generate appropriate agent responses. It probes intent understanding, table-level grounding, and robustness to temporal distribution shifts. Use when the user wants to benchmark on StatCan Dialogue Dataset, or asks about evaluating this task. Reports recall@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.01412
  bibtex_key: lu2023statcan
  confidence: high
---

# statcan-dialogue-eval

> The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents — Lu et al. (2023) (arXiv:2304.01412, 2023)

## What this evaluates

Evaluates a model's ability to retrieve relevant statistical data tables from a large corpus based on conversational dialogue history, and its ability to generate appropriate agent responses. It probes intent understanding, table-level grounding, and robustness to temporal distribution shifts.

## Datasets

- **StatCan Dialogue Dataset** — total 19379; splits: train (-1), val (-1), test (-1)

## Metrics

- `recall@10` **(primary)** — range: [0, 1]
  - Fraction of test instances where the ground truth table title appears in the top-10 retrieved candidates.
- `Title Acc.` — range: [0, 1]
  - Exact match accuracy where the generated response exactly matches the ground truth table title.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence overlap between generated and reference responses, scaled to [0, 1].

## Input / output format

**Input**: Dialogue history (user-agent turns) and optional table metadata (title, member items, footnotes, basic info).

**Output**: For retrieval: a ranked list of candidate table titles. For generation: a single text response string.

## Scoring recipe

```python
def calc_recall_at_k(preds, golds, k=10):
    hits = sum(1 for p, g in zip(preds, golds) if g in p[:k])
    return hits / len(golds)

def calc_title_acc(preds, golds):
    correct = sum(1 for p, g in zip(preds, golds) if p.strip() == g.strip())
    return correct / len(golds)
```

## Common pitfalls

- Models exhibit significant performance drops on the test split due to temporal drift (new tables and shifted conversation topics not seen in training).
- Context length limits cause truncation of table metadata, drastically hurting retrieval recall when full tables are used instead of concise metadata.
- French split performance is consistently lower due to smaller dataset size, which may confound cross-lingual comparisons.

## Evidence (verbatim from paper)

> We observe in Table 9 that DPR outperforms TAPAS and TAPAS-NQ by respectively 23.0% and 15.1% in test recall@10. Moreover, TAPAS-NQ achieves a better performance when it only retrieves the title and member items instead of the full table (p = 0.016), likely due to repetitions and truncation due to context size limits.

## Citation

```bibtex
@misc{lu2023statcan,
  title={The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents},
  author={Lu et al. (2023)},
  year={2023},
  note={arXiv:2304.01412}
}
```

- arXiv: 2304.01412