Dsp Eval

---
name: dsp-eval
description: Evaluates retrieval-augmented language models on open-domain, multi-hop, and conversational question answering by testing their ability to dynamically search for evidence, bootstrap in-context demonstrations, and generate accurate answers without fine-tuning. Use when the user wants to benchmark on Open-SQuAD, HotPotQA, QReCC, or asks about evaluating this task. Reports EM.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.14024
  bibtex_key: khattab2022demonstratesearchpredict
  confidence: high
---

# dsp-eval

> Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP — Khattab et al. (2022) (arXiv:2212.14024, 2022)

## What this evaluates

Evaluates retrieval-augmented language models on open-domain, multi-hop, and conversational question answering by testing their ability to dynamically search for evidence, bootstrap in-context demonstrations, and generate accurate answers without fine-tuning.

## Datasets

- **Open-SQuAD** — total ?; splits: dev (1000), test (-1)
- **HotPotQA** — total ?; splits: val (1000), train (-1)
- **QReCC** — total ?; splits: val (400), train (-1)

## Metrics

- `EM` **(primary)** — range: [0, 1]
  - Exact match accuracy: 1 if the predicted answer string exactly matches the gold answer string, 0 otherwise.
- `F1` — range: [0, 1]
  - Token-level F1 score computed as the harmonic mean of precision and recall between the predicted and gold answer tokens.
- `nF1` — range: [0, 1]
  - Novel F1 score: token-level F1 overlap between system response and ground truth, discounting common stopwords and terms present in the question or earlier conversation turns.

## Input / output format

**Input**: A question string (for open-domain and multi-hop QA) or a sequence of conversation turns (for conversational QA), provided alongside up to 16 randomly sampled training examples and retrieved Wikipedia passages.

**Output**: A single predicted answer string (or conversational response), generated via greedy decoding or selected via self-consistency voting over 20 sampled reasoning chains.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task='qa'):
    em_scores = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    f1_scores = [token_f1(p, g) for p, g in zip(predictions, golds)]
    if task == 'convqa':
        nf1_scores = [novel_f1(p, g, question) for p, g, question in zip(predictions, golds, questions)]
        return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1_scores)/len(f1_scores), 'nF1': sum(nf1_scores)/len(nf1_scores)}
    return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1_scores)/len(f1_scores)}
```

## Common pitfalls

- Validation and test sets are subsampled to 1000 questions (or 400 conversations) rather than using the full official splits.
- Results are averaged over 5 random seeds, each using a different 16-shot training subset, introducing sampling variance not always reported.
- nF1 specifically discounts stopwords and question terms, differing from standard conversational F1 metrics.

## Evidence (verbatim from paper)

> We subsample the validation and test sets to 1000 questions (or 400 conversations, where applicable) and report average quality across five seeds where each seed fixes a single k-shot training set of examples. Table 1 reports the answer EM and F1. We report the novel-F1 metric (nF1; Paranjape et al. 2022), which computes the F1 overlap between the system response and the ground truth while discounting common stopwords and terms present in the question (or earlier questions).

## Citation

```bibtex
@misc{khattab2022demonstratesearchpredict,
  title={Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP},
  author={Khattab et al. (2022)},
  year={2022},
  note={arXiv:2212.14024}
}
```

- arXiv: 2212.14024