Moai Ml Rag

---
name: moai-ml-rag
version: 4.0.0
updated: "2025-11-19"
status: stable
stability: stable
description: Retrieval-Augmented Generation systems, vector databases, embedding strategies, and production RAG architectures for enterprise LLM applications. Use when building RAG, semantic search, or knowledge-aware AI systems.
allowed-tools:
  - Read
  - Bash
  - WebSearch
  - WebFetch
---

# Retrieval-Augmented Generation (RAG) — Enterprise

Production-grade RAG systems combining semantic search, vector databases, and LLM generation.

## Quick Start

**5-Minute RAG with LangChain**:

```python
from langchain.vectorstores import Pinecone
from langchain.embeddings import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

# 1. Create embeddings
embeddings = OpenAIEmbeddings()

# 2. Setup vector store
vectorstore = Pinecone.from_documents(
    documents, embeddings, index_name="docs"
)

# 3. Create RAG chain
qa = RetrievalQA.from_chain_type(
    llm=OpenAI(),
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3})
)

# 4. Ask questions
answer = qa.run("What is the refund policy?")
```

**Auto-triggers**: RAG, retrieval, vector search, semantic search, embedding, knowledge base, Q&A

---

## Core Concepts

### RAG vs Fine-tuning

| Aspect                | RAG                   | Fine-tuning           |
| --------------------- | --------------------- | --------------------- |
| **Knowledge Updates** | Instant (update docs) | Requires retraining   |
| **Cost**              | Low ($50-500/mo)      | High ($1000-10000)    |
| **Time to Deploy**    | Minutes               | Days/weeks            |
| **Knowledge Scope**   | Unlimited             | Fixed (training data) |
| **Hallucinations**    | Reduced (grounded)    | Can still occur       |
| **Use Case**          | QA, search, knowledge | Specialized language  |
| **Recommendation**    | Use first (80% cases) | Only if needed        |

**When to Use**:

- **RAG**: Customer support, FAQ, documentation QA, research
- **Fine-tuning**: Domain jargon mastery, custom behavior
- **Both**: Large knowledge base + custom model behavior

### The 4-Step RAG Pipeline

```
1. INDEXING (Offline)
   → Load documents
   → Split into chunks
   → Create embeddings
   → Store in vector DB

2. RETRIEVAL (Query time)
   → Embed user query
   → Search vector DB
   → Retrieve top-k results

3. RANKING (Optional)
   → Reorder by relevance
   → Filter low-confidence
   → Deduplicate

4. GENERATION (LLM)
   → Build prompt with context
   → Generate answer
   → Post-process output
```

---

## Vector Databases

### Comparison

| Database     | Type        | Best For                 | Scale     | Cost             |
| ------------ | ----------- | ------------------------ | --------- | ---------------- |
| **Pinecone** | Cloud       | Production, ease of use  | Billions  | $70+/mo          |
| **Weaviate** | Open-source | Self-hosted, flexibility | Millions  | Free (self-host) |
| **Milvus**   | Open-source | Large-scale, distributed | Billions  | Free (self-host) |
| **Chroma**   | Local       | Development, prototyping | Thousands | Free             |
| **FAISS**    | Library     | Embedding-only, research | Millions  | Free             |

**Selection Guide**:

- **Development**: Chroma (local, fast setup)
- **Production**: Pinecone (managed, reliable)
- **Self-hosted**: Weaviate (open-source, feature-rich)
- **Large-scale**: Milvus (distributed, scalable)

### Pinecone Example

```python
import pinecone
from sentence_transformers import SentenceTransformer

# Initialize
pinecone.init(api_key="your-key", environment="us-west1-gcp")
index = pinecone.Index("docs")

# Embed and store
model = SentenceTransformer('all-MiniLM-L6-v2')
embedding = model.encode("Python is a programming language")
index.upsert([("doc1", embedding.tolist(), {"text": "..."})])

# Search
results = index.query(embedding.tolist(), top_k=3, include_metadata=True)
```

---

## Embedding Models

### Model Comparison (2025)

| Model                | Dimension | Speed  | Accuracy  | Size  | Use Case         |
| -------------------- | --------- | ------ | --------- | ----- | ---------------- |
| **all-MiniLM-L6-v2** | 384       | Fast   | Good      | 22MB  | Development      |
| **bge-base-en-v1.5** | 768       | Medium | Excellent | 438MB | Production (EN)  |
| **multilingual-e5**  | 768       | Medium | Very Good | 460MB | Multilingual     |
| **OpenAI ada-002**   | 1536      | Fast   | Excellent | API   | Cloud production |
| **Cohere embed-v3**  | 1024      | Fast   | Excellent | API   | Enterprise       |

**Selection**:

- **Development**: all-MiniLM-L6-v2 (fast, small)
- **Production**: bge-large-en-v1.5 or OpenAI ada-002
- **Multilingual**: multilingual-e5-base (100+ languages)
- **Balanced**: bge-base-en-v1.5

### Usage Example

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

# Encode documents
docs = ["Python is great", "Java is popular"]
embeddings = model.encode(docs, batch_size=32)

# Similarity search
query_embedding = model.encode("programming languages")
similarities = model.similarity(query_embedding, embeddings)
```

---

## Chunking Strategies

### Best Practices

**Chunk Size**:

- **Default**: 512 tokens (~400 words)
- **Short answers**: 256 tokens
- **Long context**: 1024 tokens

**Overlap**:

- **Recommended**: 25% overlap (128 tokens for 512-token chunks)
- **Prevents**: Context loss at boundaries

**Example**:

```python
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=128,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = splitter.split_text(long_document)
```

---

## RAG Evaluation Metrics

### Key Metrics

| Metric                | Description                                       | Target |
| --------------------- | ------------------------------------------------- | ------ |
| **Hit Rate**          | % queries with relevant doc retrieved             | >90%   |
| **MRR**               | Mean Reciprocal Rank (position of first relevant) | >0.8   |
| **NDCG@k**            | Normalized DCG (ranking quality)                  | >0.85  |
| **Latency**           | Query → Answer time                               | <500ms |
| **Context Relevance** | Retrieved docs actually used in answer            | >80%   |

### Evaluation Example

```python
def calculate_hit_rate(retrieved, relevant):
    """Hit Rate: % of queries where relevant doc was found"""
    hits = len(set(retrieved) & set(relevant))
    return hits / len(relevant) if relevant else 0

def calculate_mrr(retrieved, relevant_doc):
    """Mean Reciprocal Rank: 1/(position of first relevant)"""
    for i, doc in enumerate(retrieved):
        if doc == relevant_doc:
            return 1.0 / (i + 1)
    return 0.0
```

---

## Advanced Patterns

### Hybrid Retrieval (Dense + Sparse)

Combine vector search (semantic) with BM25 (keyword):

```python
from langchain.retrievers import BM25Retriever, EnsembleRetriever

# Dense retrieval
vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

# Sparse retrieval
bm25_retriever = BM25Retriever.from_documents(documents)

# Ensemble (0.5 weight each)
ensemble = EnsembleRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    weights=[0.5, 0.5]
)

results = ensemble.get_relevant_documents("query")
```

### Re-ranking with Cross-Encoders

Improve retrieval quality:

```python
from sentence_transformers import CrossEncoder

# Initial retrieval (fast, lower quality)
initial_results = vectorstore.similarity_search(query, k=20)

# Re-rank (slow, higher quality)
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
pairs = [[query, doc.page_content] for doc in initial_results]
scores = reranker.predict(pairs)

# Sort and return top-k
reranked = sorted(zip(initial_results, scores), key=lambda x: x[1], reverse=True)
top_results = [doc for doc, score in reranked[:5]]
```

---

## Performance Optimization

### Caching Strategy

```python
import redis
from functools import lru_cache

r = redis.Redis(host='localhost', port=6379, db=0)

def cached_embedding(text):
    """Cache embeddings in Redis"""
    key = f"emb:{hash(text)}"
    cached = r.get(key)

    if cached:
        return pickle.loads(cached)

    embedding = model.encode(text)
    r.setex(key, 86400, pickle.dumps(embedding))  # 24h TTL
    return embedding
```

### Streaming Responses

```python
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

qa = RetrievalQA.from_chain_type(
    llm=OpenAI(streaming=True, callbacks=[StreamingStdOutCallbackHandler()]),
    retriever=retriever
)

# Streams tokens as they're generated
qa.run("What is RAG?")
```

---

## Production Best Practices

✅ **DO**:

- Start with simple RAG, iterate
- Evaluate retrieval quality (hit rate, MRR)
- Monitor latency and costs
- Cache embeddings and responses
- Use hybrid retrieval for better coverage
- Implement re-ranking for quality
- Log queries for improvement

❌ **DON'T**:

- Assume retrieval always works
- Skip evaluation metrics
- Use only vector search (add BM25)
- Ignore chunk size tuning
- Store sensitive data unencrypted

---

## Common Issues & Solutions

| Issue                 | Solution                                                          |
| --------------------- | ----------------------------------------------------------------- |
| Low retrieval quality | Try hybrid retrieval, adjust chunk size, use re-ranking           |
| Slow queries          | Cache embeddings, use faster embedding models, optimize vector DB |
| High costs            | Use open-source embeddings, cache responses, batch requests       |
| Hallucinations        | Improve retrieval, use stricter prompts, add confidence scores    |

---

## Advanced Topics

For detailed implementation patterns, see:

- **[examples.md](examples.md)**: Complete RAG implementations, authentication flows, multi-hop reasoning
- **[reference.md](reference.md)**: API references, vector DB setup, production deployment

**Related Skills**:

- `moai-ml-llm-fine-tuning`: LLM fine-tuning patterns
- `moai-domain-ml`: ML best practices
- `moai-essentials-perf`: Performance optimization

---

**Key Libraries**: LangChain 0.2+, LlamaIndex 0.10+, Pinecone 3.0+, sentence-transformers 3.0+

**Version**: 4.0.0  
**Last Updated**: 2025-11-19  
**Status**: Production Ready