Smd Few Shot Eval

---
name: smd-few-shot-eval
description: Evaluates data efficiency and response generation quality in goal-oriented dialogue systems under few-shot conditions. It measures how well a model can generate contextually appropriate and entity-accurate responses using only a small fraction of in-domain dialogue data. Use when the user wants to benchmark on SMD, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.01302
  bibtex_key: shalyminov2019dataefficient
  confidence: high
---

# smd-few-shot-eval

> Data-Efficient Goal-Oriented Conversation with Dialogue Knowledge Transfer Networks — Shalyminov et al. (2019) (arXiv:1910.01302, 2019)

## What this evaluates

Evaluates data efficiency and response generation quality in goal-oriented dialogue systems under few-shot conditions. It measures how well a model can generate contextually appropriate and entity-accurate responses using only a small fraction of in-domain dialogue data.

## Datasets

- **SMD** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 100] percent
  - Standard n-gram overlap score between predicted and gold responses, computed to conform with the Zhao and Eskenazi (2018) setup. It measures lexical similarity but does not account for semantic equivalence.
- `Entity F1` — range: [0, 1]
  - F1 score measuring the precision and recall of correctly predicted entities in the response compared to the gold response. It evaluates goal-oriented accuracy rather than pure language generation.

## Input / output format

**Input**: Dialogue context consisting of alternating user and system turns.

**Output**: A single predicted response string.

## Scoring recipe

```python
# Evaluate across multiple data percentages (e.g., 1%, 5%, 10%, 50%)
# Average over 10 runs with 10 different sets of seed dialogues per run
bleu_scores = []
entity_f1_scores = []
for run in range(10):
    for seed_set in range(10):
        pred = model.generate(context)
        bleu_scores.append(compute_bleu(gold, pred))
        entity_f1_scores.append(compute_entity_f1(gold, pred))
return mean(bleu_scores), mean(entity_f1_scores)
```

## Common pitfalls

- High variance in reported results due to the evaluation protocol averaging over 10 runs with 10 different random seed dialogues.
- BLEU may penalize semantically correct responses that use different wording than the gold reference, which is common in open-ended dialogue generation.

## Evidence (verbatim from paper)

> We use BLEU as one of the main evaluation metrics in this paper — we do it in order to fully conform with the setup of Zhao and Eskenazi (2018) which we base our work on. But while being widely adopted as a general-purpose language generation metric, BLEU might not be sufficient in the dialogue setting (see Novikova et al. (2017) for a review). Specifically, we have observed several cases where the model would produce an overall grammatical response with the correct dialogue intent (e.g. "You are welcome! Anything else?"), but BLEU would output a lower score for it due to word mismatch (e.g. "You're welcome!"; see more examples in Table 4).

## Citation

```bibtex
@misc{shalyminov2019dataefficient,
  title={Data-Efficient Goal-Oriented Conversation with Dialogue Knowledge Transfer Networks},
  author={Shalyminov et al. (2019)},
  year={2019},
  note={arXiv:1910.01302}
}
```

- arXiv: 1910.01302