Open Domain Qa Eval

---
name: open-domain-qa-eval
description: Evaluates open-domain question answering systems on their ability to retrieve relevant context and generate accurate answers across straightforward (OLTP) and synthesis-heavy (OLAP) queries. It measures factual correctness against reference answers and assesses multi-dimensional answer quality (comprehensiveness, diversity, empowerment) for open-ended questions. Use when the user wants to benchmark on HotPotQA, MSMarco, Microsoft Earnings Call Transcripts, Kevin Scott Podcast Transcripts, or asks about evaluating this task. Reports LLM-as-a-judge accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.02922
  bibtex_key: [REDACTED]
  confidence: high
---

# open-domain-qa-eval

> Optimizing open-domain question answering with graph-based retrieval augmented generation — Cahoon et al. (2025) (arXiv:2503.02922, 2025)

## What this evaluates

Evaluates open-domain question answering systems on their ability to retrieve relevant context and generate accurate answers across straightforward (OLTP) and synthesis-heavy (OLAP) queries. It measures factual correctness against reference answers and assesses multi-dimensional answer quality (comprehensiveness, diversity, empowerment) for open-ended questions.

## Datasets

- **HotPotQA** — total 5491; splits: dev (5491)
- **MSMarco** — total 1000; splits: test (1000)
- **Microsoft Earnings Call Transcripts** — total 40; splits: test (40)
- **Kevin Scott Podcast Transcripts** — total 125; splits: test (125)

## Metrics

- `LLM-as-a-judge accuracy` **(primary)** — range: [0, 1]
  - Binary correctness score for OLTP queries where the LLM judge outputs YES (correct) or NO (incorrect) by comparing the generated answer to a reference answer. For OLAP queries, it aggregates qualitative scores across Comprehensiveness, Diversity, and Empowerment dimensions.

## Input / output format

**Input**: Open-domain question/query and retrieved context (from graph, vector, or hybrid retrieval pipelines).

**Output**: Generated natural language answer. For evaluation, the LLM judge outputs a binary YES/NO for OLTP or qualitative scores for OLAP dimensions.

## Scoring recipe

```python
def evaluate(predictions, golds, query_type):
    if query_type == 'OLTP':
        correct = [1 if judge(prediction, gold) == 'YES' else 0 for prediction, gold in zip(predictions, golds)]
        return sum(correct) / len(correct)
    else:  # OLAP
        scores = [judge_comprehensive(p) + judge_diverse(p) + judge_empower(p) for p in predictions]
        return sum(scores) / len(scores)
```

## Common pitfalls

- OLAP queries lack fixed ground truth, making correctness subjective and requiring multi-dimensional scoring rather than exact match.
- LLM-as-a-judge introduces variability and potential bias compared to human experts, despite reported 80%+ agreement.
- HotPotQA requires filtering out queries directly answerable without context to prevent data leakage and ensure retrieval is actually tested.

## Evidence (verbatim from paper)

> To assess answer quality, we apply LLM-as-a-judge for both OLTP and OLAP-style benchmarks: OLTP Evaluation: Since answers have clear ground truths, we compare model-generated responses against reference answers, achieving 99%+ agreement. To ensure consistency, we apply a logit bias, restricting outputs to “YES” (correct) or “NO” (incorrect). OLAP Evaluation: As no fixed ground truth exists, we adopt the GraphRAG evaluation framework, assessing responses on: Comprehensiveness: Depth and thoroughness of information. Diversity: Inclusion of multiple perspectives. Empowerment: How well the answer informs decision-making.

## Citation

```bibtex
@misc{cahoon2025optimizing,
  title={Optimizing open-domain question answering with graph-based retrieval augmented generation},
  author={Cahoon et al. (2025)},
  year={2025},
  note={arXiv:2503.02922}
}
```

- arXiv: 2503.02922