Few Shot Bot Eval

---
name: few-shot-bot-eval
description: Evaluates prompt-based large language models on 15 diverse dialogue tasks, including response generation, conversational parsing, and skill selection, using a few-shot learning setup without fine-tuning. The protocol tests the model's ability to dynamically select the most appropriate task prompt based on dialogue history and generate accurate responses or parses. Use when the user wants to benchmark on Persona Chat, Empathetic Dialogues (ED), Wizard of Wikipedia (WoW), Image Chat (IC), Wizard of Internet (WIT), Controlled Generation (CG-IC), Multi-Session Chat (MSC), DailyDialogue (DD), Stanford Multidomain Dialogue (SMD), DialKG, or asks about evaluating this task. Reports perplexity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.08118
  bibtex_key: madotto2021fewshotbot
  confidence: low
---

# few-shot-bot-eval

> Few-Shot Bot: Prompt-Based Learning for Dialogue Systems — Andrea Madotto et al. (arXiv:2110.08118, 2021)

## What this evaluates

Evaluates prompt-based large language models on 15 diverse dialogue tasks, including response generation, conversational parsing, and skill selection, using a few-shot learning setup without fine-tuning. The protocol tests the model's ability to dynamically select the most appropriate task prompt based on dialogue history and generate accurate responses or parses.

## Datasets

- **Persona Chat** — total ?; splits: test (-1)
- **Empathetic Dialogues (ED)** — total ?; splits: test (-1)
- **Wizard of Wikipedia (WoW)** — total ?; splits: test (-1)
- **Image Chat (IC)** — total ?; splits: test (-1)
- **Wizard of Internet (WIT)** — total ?; splits: test (-1)
- **Controlled Generation (CG-IC)** — total ?; splits: test (-1)
- **Multi-Session Chat (MSC)** — total ?; splits: test (-1)
- **DailyDialogue (DD)** — total ?; splits: test (-1)
- **Stanford Multidomain Dialogue (SMD)** — total ?; splits: test (-1)
- **DialKG** — total ?; splits: test (-1)

## Metrics

- `perplexity` **(primary)** — range: other
  - Computed over the prompt and candidate response to dynamically select the most appropriate skill for a given dialogue history. Lower perplexity indicates a better prompt-skill match. (Note: Specific metrics for response generation and conversational parsing are not detailed in the provided text.)

## Input / output format

**Input**: Dialogue history/context paired with a few-shot prompt template containing task-specific examples for each candidate skill.

**Output**: Task-specific output: natural language response, parsed dialogue act/intent, or selected skill identifier.

## Scoring recipe

```python
# Pseudo-code for skill selection and task evaluation
# Note: Generation/parsing metrics are not specified in the provided text.

def evaluate_skill_selection(dialogue_history, candidate_skills, model):
    best_skill = None
    min_ppl = float('inf')
    for skill in candidate_skills:
        prompt = build_prompt(dialogue_history, skill, few_shot_examples)
        ppl = model.compute_perplexity(prompt)
        if ppl < min_ppl:
            min_ppl = ppl
            best_skill = skill
    return best_skill, min_ppl

def evaluate_generation_or_parsing(predictions, gold):
    # Task-specific metrics (e.g., BLEU, ROUGE, Accuracy) not detailed in provided text
    # Placeholder for standard evaluation
    return compute_task_metric(predictions, gold)
```

## Common pitfalls

- Prompt ordering and example selection significantly impact few-shot performance and can introduce bias.
- Perplexity-based skill selection may not align perfectly with human judgment or downstream task accuracy.
- Zero-shot/few-shot LLMs often show high variance across random seeds, temperature settings, or prompt formatting.

## Evidence (verbatim from paper)

> Prompt-based few-shot learning enables large language models to achieve performance comparable to fully trained state-of-the-art dialogue models across 15 diverse tasks—including knowledge-grounded response generation, dialogue parsing, and stylistic control—without any fine-tuning. A novel perplexity-based few-shot skill selector dynamically chooses the most appropriate prompt for a given dialogue history, enabling an end-to-end, zero-training chatbot (Few-Shot Bot) that autonomously selects skills, retrieves external knowledge, and generates human-like responses using only a few examples per task.

## Citation

```bibtex
@misc{madotto2021fewshotbot,
  title={Few-Shot Bot: Prompt-Based Learning for Dialogue Systems},
  author={Andrea Madotto et al.},
  year={2021},
  note={arXiv:2110.08118}
}
```

- arXiv: 2110.08118