Dora Eval

---
name: dora-eval
description: Evaluates RAG-based question answering systems on defense-domain documents, measuring both retrieval effectiveness and end-to-end QA performance including task success, faithfulness, and generation quality. Use when the user wants to benchmark on DoRA, or asks about evaluating this task. Reports task-success.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17943
  bibtex_key: doan2026dora
  confidence: high
---

# dora-eval

> Domain-oriented RAG Assessment (DoRA): Synthetic Benchmarking for RAG-based Question Answering on Defense Documents — Doan et al. (2026) (arXiv:2604.17943, 2026)

## What this evaluates

Evaluates RAG-based question answering systems on defense-domain documents, measuring both retrieval effectiveness and end-to-end QA performance including task success, faithfulness, and generation quality.

## Datasets

- **DoRA** — total 6500; splits: train (4635), eval (515)

## Metrics

- `task-success` **(primary)** — range: percent
  - Percentage of questions answered correctly according to reference answers or expert criteria.
- `faithfulness` — range: [0, 1]
  - RAGEval-based metric measuring completeness, hallucination, and irrelevance of generated answers relative to retrieved evidence.
- `Tok. F1` — range: [0, 1]
  - F1 score computed over token-level overlap between predicted and reference answers.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence recall/precision between predicted and reference answers.
- `BLEU` — range: [0, 1]
  - Bilingual evaluation understudy score measuring n-gram precision with brevity penalty.

## Input / output format

**Input**: Question q concatenated with retrieved evidence context ctx(E) (or gold context E for oracle evaluation).

**Output**: Generated answer a.

## Scoring recipe

```python
def compute_metrics(predictions, references, contexts):
    token_f1 = [token_f1_score(p, r) for p, r in zip(predictions, references)]
    rouge_l = [rouge_l_score(p, r) for p, r in zip(predictions, references)]
    bleu = [bleu_score(p, r) for p, r in zip(predictions, references)]
    task_success = [is_correct(p, r) for p, r in zip(predictions, references)]
    faithfulness = [rag_faithfulness(p, c) for p, c in zip(predictions, contexts)]
    return {
        'token_f1': mean(token_f1),
        'rouge_l': mean(rouge_l),
        'bleu': mean(bleu),
        'task_success': mean(task_success),
        'faithfulness': mean(faithfulness)
    }
```

## Common pitfalls

- Retrieval error often dominates performance, masking model generation quality; fixed retriever settings should be reported.
- Faithfulness metrics (completeness, hallucination, irrelevance) require careful alignment with retrieved evidence, not just reference answers.
- Synthetic data generation may introduce intent mismatch or contamination if seed documents overlap with training corpora.

## Evidence (verbatim from paper)

> Results in Table 2 show that under a fixed retrieval setting, general-purpose LLMs cluster closely on both task-success and faithfulness, suggesting that retrieval and grounding constraints dominate performance in this private, defense-domain corpus.

## Citation

```bibtex
@misc{doan2026dora,
  title={Domain-oriented RAG Assessment (DoRA): Synthetic Benchmarking for RAG-based Question Answering on Defense Documents},
  author={Doan et al. (2026)},
  year={2026},
  note={arXiv:2604.17943}
}
```

- arXiv: 2604.17943