Wikichat Simulated Dialogue Eval

---
name: wikichat-simulated-dialogue-eval
description: Evaluates the factual accuracy, conversational quality, and latency of knowledge-grounded chatbots in simulated multi-turn dialogues across head, tail, and recent knowledge domains. Use when the user wants to benchmark on Simulated Dialogues (WikiChat), or asks about evaluating this task. Reports factual_accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14292
  bibtex_key: semnani2023wikichat
  confidence: high
---

# wikichat-simulated-dialogue-eval

> WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia — Semnani et al. (2023) (arXiv:2305.14292, 2023)

## What this evaluates

Evaluates the factual accuracy, conversational quality, and latency of knowledge-grounded chatbots in simulated multi-turn dialogues across head, tail, and recent knowledge domains.

## Datasets

- **Simulated Dialogues (WikiChat)** — total ?; splits: head (-1), tail (-1), recent (-1); repo https://github.com/stanford-oval/WikiChat

## Metrics

- `factual_accuracy` **(primary)** — range: percent
  - Percentage of generated claims supported by the Wikipedia knowledge corpus. Calculated as (number of claims verified as true by majority of 3 crowdworkers) / (total claims generated).

## Input / output format

**Input**: Simulated user prompt containing only the title and first sentence of a Wikipedia article, plus the conversation history up to the current turn.

**Output**: Chatbot response text per turn.

## Scoring recipe

```python
total_claims = 0
supported_claims = 0
for response in bot_responses:
    claims = extract_claims(response)
    for claim in claims:
        total_claims += 1
        judgments = get_crowdworker_judgments(claim, wikipedia_corpus) # 3 workers
        if majority_agree(claim, judgments):
            supported_claims += 1
return supported_claims / total_claims
```

## Common pitfalls

- Evaluating only on head knowledge masks severe hallucination in tail and recent knowledge domains.
- Factuality is measured per-claim rather than per-response, requiring accurate claim extraction before evaluation.
- Simulated users driven by GPT-4 may not perfectly replicate human knowledge gaps or conversational behavior.

## Evidence (verbatim from paper)

> We define the factual accuracy of a chatbot to be the percentage of claims the bot makes in a given dialogue set, that are supported by the knowledge corpus. As mentioned in Section 5.2, this is done by obtaining per-claim judgments of factuality from crowdworkers. We obtain 3 judgements for each of the 5974 claims our chatbots output in total.

## Citation

```bibtex
@misc{semnani2023wikichat,
  title={WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia},
  author={Semnani et al. (2023)},
  year={2023},
  note={arXiv:2305.14292}
}
```

- arXiv: 2305.14292