Qa Quality Eval

---
name: qa-quality-eval
description: Evaluates the quality of generated question-answer pairs in an agricultural domain context. It probes a model's ability to produce relevant, accurate, diverse, and fluent Q&A content under varying context conditions. Use when the user wants to benchmark on Agricultural Q&A dataset, or asks about evaluating this task. Reports Relevance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.08406
  bibtex_key: balaguer2024ragvsfinetuning
  confidence: high
---

# qa-quality-eval

> RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture — Balaguer et al. (2024) (arXiv:2401.08406, 2024)

## What this evaluates

Evaluates the quality of generated question-answer pairs in an agricultural domain context. It probes a model's ability to produce relevant, accurate, diverse, and fluent Q&A content under varying context conditions.

## Datasets

- **Agricultural Q&A dataset** — total ?; splits: test (-1)

## Metrics

- `Relevance` **(primary)** — range: [1, 5]
  - LLM-judged score on a 1-5 scale assessing how pertinent the generated answer is to the question.
- `Coverage` — range: [1, 5]
  - LLM-judged score on a 1-5 scale measuring how comprehensively the answer addresses the question and covers relevant source text sections.
- `Diversity` — range: other
  - Computed using Word Mover's Distance (WMD) or KL divergence to measure semantic dissimilarity between generated questions and source text; lower scores indicate higher diversity.
- `Overlap` — range: other
  - Computed using KL divergence or WMD to measure semantic similarity between generated questions and source text; higher scores indicate more overlap.
- `Fluency` — range: [1, 5]
  - LLM-judged score on a 1-5 scale evaluating the grammatical correctness and natural flow of the generated answer.
- `Correctness` — range: [1, 5]
  - LLM-judged categorical score (e.g., Correct, Partially Correct, Incorrect) mapped to a 1-5 scale based on factual alignment with the reference answer.
- `Succinctness` — range: [1, 5]
  - LLM-judged score on a 1-5 scale penalizing verbose or overly detailed responses that go beyond the scope of the question.

## Input / output format

**Input**: Question, generated answer, reference answer, and optional context (no context, location/state context, or external geographic context).

**Output**: A numerical score from 1 to 5 for each metric, accompanied by a textual explanation justifying the score.

## Scoring recipe

```python
def evaluate_qa(question, answer, reference, context, metric):
    if metric in ['Overlap', 'Diversity']:
        return compute_wmd_or_kl(question, context)
    else:
        prompt = f'Score the {metric} of this answer (1-5): Q: {question} A: {answer} Ref: {reference}'
        return llm_judge_score(prompt, scale=5)

# Aggregate across all instances
final_score = mean([evaluate_qa(q, a, r, c, m) for q, a, r, c in dataset])
```

## Common pitfalls

- LLM judges tend to penalize concise answers if they lack elaboration, as seen in the Succinctness metric where verbose but relevant answers score lower.
- Diversity and Overlap require specific distance metrics (WMD/KL divergence) rather than simple LLM scoring, which can lead to implementation inconsistencies if not strictly followed.
- Context setup variations (no context vs. external context) drastically alter prompt length and model behavior, making direct metric comparisons across setups sensitive to formatting.

## Evidence (verbatim from paper)

> We used several metrics (defined on Section 4.1) to assess their quality, including Relevance, Global Relevance, Coverage, Overlap, Diversity, Details, and Fluency. The LLMs assessed the Q&A pairs on various metrics, scoring each on a scale from 1 to 5. For certain metrics such as Overlap and Diversity, we incorporated intricate methods like Kullback-Leibler (KL) divergence and Word Mover's Distance (WMD) to measure the semantic similarity between the source text and the questions generated.

## Citation

```bibtex
@misc{balaguer2024ragvsfinetuning,
  title={RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture},
  author={Balaguer et al. (2024)},
  year={2024},
  note={arXiv:2401.08406}
}
```

- arXiv: 2401.08406