Hirag Eval

---
name: hirag-eval
description: Evaluates the answer quality of Retrieval-Augmented Generation (RAG) systems across specialized domains. It measures how well generated responses address queries in terms of comprehensiveness, empowerment, diversity, and overall performance using pairwise LLM-as-a-judge comparisons. Use when the user wants to benchmark on UltraDomain, or asks about evaluating this task. Reports win rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.10150
  bibtex_key: huang2025hirag
  confidence: high
---

# hirag-eval

> Retrieval-Augmented Generation with Hierarchical Knowledge — Haoyu Huang et al. (2025) (arXiv:2503.10150, 2025)

## What this evaluates

Evaluates the answer quality of Retrieval-Augmented Generation (RAG) systems across specialized domains. It measures how well generated responses address queries in terms of comprehensiveness, empowerment, diversity, and overall performance using pairwise LLM-as-a-judge comparisons.

## Datasets

- **UltraDomain** — total ?; splits: test (-1)

## Metrics

- `win rate` **(primary)** — range: percent
  - Percentage of instances where a method's answer is judged superior to a baseline's answer by an LLM judge across four dimensions: Comprehensiveness, Empowerment, Diversity, and Overall.

## Input / output format

**Input**: Query, answer from method A, answer from method B (order alternated for fairness)

**Output**: LLM judge selects the superior answer per dimension (or declares a tie)

## Scoring recipe

```python
wins = 0
total = 0
for query, ans_A, ans_B in dataset:
    judge_A = llm_judge(ans_A, ans_B, dimensions)
    judge_B = llm_judge(ans_B, ans_A, dimensions) # order alternated
    if judge_A > judge_B: wins += 1
    total += 1
win_rate = (wins / total) * 100
```

## Common pitfalls

- LLM-as-a-judge can exhibit position bias, mitigated here by alternating answer order.
- Dimensions like 'Empowerment' and 'Diversity' are subjective and depend heavily on the prompt instructions given to the judge LLM.
- Win rates are calculated pairwise against each baseline separately, not as a single leaderboard ranking.

## Evidence (verbatim from paper)

> We used the win rate to compare different methods, which indicates the percentage of instances that a method generates higher-quality answers compared to another method as judged by the LLM. We utilized GPT-4o as the evaluation model to judge which method generates a superior answer for each query for the following four dimensions: (1) Comprehensiveness: how thoroughly does the answer address the question, covering all relevant aspects and details? (2) Empowerment: how effectively does the answer provide actionable insights or solutions that empower the user to take meaningful steps? (3) Diversity: how well does the answer incorporate a variety of perspectives, approaches, or solutions to the problem? (4) Overall: how does the answer perform overall, considering comprehensiveness, empowerment, diversity, and any other relevant factors?

## Citation

```bibtex
@misc{huang2025hirag,
  title={Retrieval-Augmented Generation with Hierarchical Knowledge},
  author={Haoyu Huang et al. (2025)},
  year={2025},
  note={arXiv:2503.10150}
}
```

- arXiv: 2503.10150