Rag Tech Docs Eval

---
name: rag-tech-docs-eval
description: Evaluates how chunking strategies, embedding models, and retrieval thresholds affect RAG performance on technical IEEE documents. Probes the impact of sentence length, keyword position, and acronym handling on retrieval relevance and generator hallucination. Use when the user wants to benchmark on IEEE Wireless LAN MAC/PHY & Battery Glossary, or asks about evaluating this task. Reports qualitative observation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.00657
  bibtex_key: soman2024observations
  confidence: medium
---

# rag-tech-docs-eval

> Observations on Building RAG Systems for Technical Documents — Soman et al. (2024) (arXiv:2404.00657, 2024)

## What this evaluates

Evaluates how chunking strategies, embedding models, and retrieval thresholds affect RAG performance on technical IEEE documents. Probes the impact of sentence length, keyword position, and acronym handling on retrieval relevance and generator hallucination.

## Datasets

- **IEEE Wireless LAN MAC/PHY & Battery Glossary** — total ?; splits: test (42)

## Metrics

- `qualitative observation` **(primary)** — range: other
  - No formal numerical formula; evaluation relies on qualitative assessment of retrieval relevance and LLM generation quality (e.g., correctness, hallucination) across 42 domain-sourced queries.

## Input / output format

**Input**: Technical document text (full paragraphs, split terms/definitions, or table-excluded documents) paired with a natural language question.

**Output**: Retrieved document chunks and an LLM-generated answer.

## Scoring recipe

```python
def evaluate_rag(doc_chunks, question, llm, embedding_model):
    q_emb = embedding_model.encode(question)
    c_embs = [embedding_model.encode(c) for c in doc_chunks]
    scores = [cosine(q_emb, e) for e in c_embs]
    top_chunks = retrieve_top_k(doc_chunks, scores, k=5)
    prompt = build_prompt(question, top_chunks)
    answer = llm.generate(prompt)
    # Qualitative assessment per paper protocol
    relevance = assess_relevance(top_chunks, question)
    hallucination = check_acronym_expansion(answer)
    return relevance, hallucination
```

## Common pitfalls

- Assuming higher similarity scores guarantee correct answers (thresholding fails to correlate with correctness)
- Treating long sentences (>200 words) as standard chunks (distorts embeddings and causes spurious matches)
- Ignoring acronym handling in definitions (leads to hallucinated expansions in generator output)

## Evidence (verbatim from paper)

> We evaluate on multiple questions and report on selected questions to substantiate our observations.

## Citation

```bibtex
@misc{soman2024observations,
  title={Observations on Building RAG Systems for Technical Documents},
  author={Soman et al. (2024)},
  year={2024},
  note={arXiv:2404.00657}
}
```

- arXiv: 2404.00657