Llm Engineering

---
name: llm-engineering
description: "Use when designing, implementing, auditing, and hardening llm engineering server logic, APIs, background jobs, and error boundaries."
version: 6.0.0
last-updated: 2026-09-29
skills:
  - ai-prompt-injection-defense
  - advanced-rag-pipelines
  - data-validation-schemas
tools: Read, Grep, Glob, Bash, Edit, Write
scripts-binding:
  - .agent/scripts/lint_runner.js
  - .agent/scripts/verify_all.js
---

# LLM Engineering — Production AI Systems Mastery

## Mandatory Pre-Flight Context Inspection
Before reading, generating, or refactoring code in the `llm-engineering` domain, inspect these 5 critical parameters:
1. **System Boundaries & Dependencies**: Verify that all required dependencies exist in target package manifests and environment paths.
2. **Runtime Context & Platform Invariants**: Confirm target platform constraints (Node.js, Browser, Mobile OS, Edge runtime) before applying APIs.
3. **Execution Guardrails**: Identify potential side-effects, state mutations, and unhandled asynchronous exceptions.
4. **Validation & Type Contracts**: Validate input data schemas and strict type constraints across all module interfaces.
5. **Observability & Proof of Execution**: Ensure execution produces tangible verification signals (terminal output, tests, metrics).


## Activation Boundaries
- **Activate when:** Use when designing, implementing, auditing, and hardening llm engineering server logic, APIs, background jobs, and error boundaries.
- **DO NOT activate when:** The task falls outside the `llm-engineering` domain or is managed by a different dedicated specialist agent.


## 🔁 Multi-Pass Execution Protocol

| Pass | Phase | Core Action | Adaptive Depth |
|:---|:---|:---|:---|
| **Pass 1** | **Understand** | Deconstruct the user's explicit objective, implicit requirements, and platform constraints. | Fast / Standard / Deep |
| **Pass 2** | **Plan** | Decompose task into smallest logical steps; map dependencies, affected files, and tool calls. | Standard / Deep |
| **Pass 3** | **Execute** | Implement solution with production-grade craft, zero placeholders, and strict typing. | All Modes |
| **Pass 4** | **Verify** | Run linters, unit tests, or compiler checks to validate structural correctness. | All Modes |
| **Pass 5** | **Attack & Falsify** | Perform adversarial search for edge-case failures, counterexamples, race conditions, and traps. | Standard / Deep |
| **Pass 6** | **Harden** | Eliminate discovered friction, optimize performance, and harden error boundaries. | Standard / Deep |
| **Pass 7** | **Quality Gate** | Enforce Verification-Before-Completion (VBC) with concrete terminal proof before finalizing. | All Modes |


---

## 🛠️ Technical Architecture & Reference Recipes

---

## Model Selection

```
Model                    │ Use Case                              │ Cost Tier
─────────────────────────┼───────────────────────────────────────┼──────────
GPT-4o                   │ Complex reasoning, vision, code       │ $$$
GPT-4o-mini              │ Classification, summaries, chat       │ $
o3-mini                  │ Deep reasoning, math, code review     │ $$
Claude 3.7 Sonnet        │ Long documents, analysis, code        │ $$$
Claude 3.5 Haiku         │ Fast responses, simple tasks          │ $
Gemini 3.1 Pro (High)    │ Large context, multimodal, code       │ $$$
Gemini 3.0 Flash         │ High throughput, cost-efficient       │ $
Llama 3.3 70B (open)     │ Self-hosted, data privacy             │ Free*
Mistral Large 2          │ European data residency, code         │ $$

* = compute costs only

Selection rules:
1. Start with the cheapest model that passes your evals
2. Upgrade only when eval scores require it
3. Use large models for complex reasoning, small for classification/routing
4. Fine-tune ONLY after prompt engineering and RAG are exhausted
5. ❌ HALLUCINATION TRAP: Model names change frequently — always verify current names
   from provider docs before hardcoding (e.g. "gpt-4o" vs "gpt-4o-2024-11-20")
```

---

## Prompt Engineering

### System Prompt Design

```typescript
const SYSTEM_PROMPT = `You are a customer support agent for Acme Corp.

## Rules
1. Answer ONLY questions about Acme products and services.
2. If you don't know the answer, say "I'll connect you with a specialist."
3. Never discuss competitors.
4. Never make up product features or pricing.
5. Keep responses under 200 words.

## Response Format
- Use bullet points for lists
- Include product links when relevant
- End with a follow-up question

## Context
Current date: ${new Date().toISOString().split('T')[0]}
User plan: {{user_plan}}
`;

// ❌ HALLUCINATION TRAP: System prompts are NOT secrets
// Users can extract system prompts with jailbreak techniques
// Never put API keys, internal URLs, or secrets in system prompts
```

### Structured Output (JSON Mode)

```typescript
import { z } from 'zod';
import OpenAI from 'openai';

const SentimentSchema = z.object({
  sentiment: z.enum(['positive', 'negative', 'neutral']),
  confidence: z.number().min(0).max(1),
  reasoning: z.string(),
  topics: z.array(z.string()),
});

// OpenAI — json_schema mode (strict = true enforces schema exactly)
async function analyzeSentiment(text: string) {
  const response = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    response_format: {
      type: 'json_schema',
      json_schema: {
        name: 'sentiment_analysis',
        strict: true,
        schema: {
          type: 'object',
          properties: {
            sentiment: { type: 'string', enum: ['positive', 'negative', 'neutral'] },
            confidence: { type: 'number' },
            reasoning: { type: 'string' },
            topics: { type: 'array', items: { type: 'string' } },
          },
          required: ['sentiment', 'confidence', 'reasoning', 'topics'],
          additionalProperties: false, // required for strict mode
        },
      },
    },
    messages: [
      { role: 'system', content: 'Analyze sentiment.' },
      { role: 'user', content: text },
    ],
  });
  const raw = JSON.parse(response.choices[0].message.content ?? '{}');
  return SentimentSchema.parse(raw); // always validate with Zod even in strict mode
}

// Gemini — response_mime_type + response_schema
import { GoogleGenerativeAI, SchemaType } from '@google/generative-ai';
const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY!);
const model = genAI.getGenerativeModel({
  model: 'gemini-2.0-flash',
  generationConfig: {
    responseMimeType: 'application/json',
    responseSchema: {
      type: SchemaType.OBJECT,
      properties: {
        sentiment: { type: SchemaType.STRING, enum: ['positive', 'negative', 'neutral'] },
        confidence: { type: SchemaType.NUMBER },
        topics: { type: SchemaType.ARRAY, items: { type: SchemaType.STRING } },
      },
      required: ['sentiment', 'confidence', 'topics'],
    },
  },
});

// ❌ HALLUCINATION TRAP: Always validate LLM JSON output with Zod/schema
// LLMs produce malformed JSON, wrong types, missing fields even with strict mode
// ❌ const result = JSON.parse(response); // trust blindly
// ✅ const result = Schema.parse(JSON.parse(response)); // validate always
```

### Function Calling / Tool Use

```typescript
const tools: OpenAI.ChatCompletionTool[] = [
  {
    type: 'function',
    function: {
      name: 'search_products',
      description: 'Search products by name, category, or price range',
      parameters: {
        type: 'object',
        properties: {
          query: { type: 'string', description: 'Search query' },
          category: { type: 'string', enum: ['electronics', 'clothing', 'home'] },
          max_price: { type: 'number', description: 'Maximum price in USD' },
        },
        required: ['query'],
      },
    },
  },
  {
    type: 'function',
    function: {
      name: 'get_order_status',
      description: 'Get the status of an order by order ID',
      parameters: {
        type: 'object',
        properties: {
          order_id: { type: 'string', description: 'The order ID (e.g., ORD-12345)' },
        },
        required: ['order_id'],
      },
    },
  },
];

// Tool execution loop
async function chatWithTools(userMessage: string) {
  const messages: OpenAI.ChatCompletionMessageParam[] = [
    { role: 'system', content: SYSTEM_PROMPT },
    { role: 'user', content: userMessage },
  ];

  let response = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages,
    tools,
  });

  // Process tool calls
  while (response.choices[0].finish_reason === 'tool_calls') {
    const toolCalls = response.choices[0].message.tool_calls ?? [];
    messages.push(response.choices[0].message);

    for (const call of toolCalls) {
      const args = JSON.parse(call.function.arguments);
      const result = await executeFunction(call.function.name, args);
      messages.push({
        role: 'tool',
        tool_call_id: call.id,
        content: JSON.stringify(result),
      });
    }

    response = await openai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages,
      tools,
    });
  }

  return response.choices[0].message.content;
}
```

---

## RAG (Retrieval-Augmented Generation)

### Pipeline

```
User Query
    ↓
[1] Embed query → vector
    ↓
[2] Search vector DB → top K chunks
    ↓
[3] (Optional) Rerank results → top N
    ↓
[4] Build prompt: system + context chunks + query
    ↓
[5] LLM generates answer with citations
    ↓
[6] Validate response (hallucination check)
```

### Chunking Strategy

```typescript
// ❌ BAD: Arbitrary character splitting
const chunks = text.match(/.{1,1000}/g); // breaks mid-sentence, mid-word

// ✅ GOOD: Semantic chunking with overlap
function chunkDocument(text: string, options: ChunkOptions = {}): Chunk[] {
  const {
    maxTokens = 512, // chunk size
    overlapTokens = 50, // overlap between chunks
    separator = '\n\n', // split on paragraph boundaries first
  } = options;

  const paragraphs = text.split(separator);
  const chunks: Chunk[] = [];
  let current = '';

  for (const para of paragraphs) {
    if (tokenCount(current + para) > maxTokens && current) {
      chunks.push({ text: current.trim(), tokens: tokenCount(current) });
      // Keep overlap from previous chunk
      const words = current.split(' ');
      current = words.slice(-overlapTokens).join(' ') + separator + para;
    } else {
      current += separator + para;
    }
  }
  if (current.trim()) chunks.push({ text: current.trim(), tokens: tokenCount(current) });

  return chunks;
}

// Chunk size guidelines:
// 256-512 tokens → precise retrieval (Q&A, support)
// 512-1024 tokens → balanced (general RAG)
// 1024-2048 tokens → broad context (summarization)
```

### Vector Store Selection

```
pgvector (PostgreSQL)  → Already using Postgres, <10M vectors, simple
Pinecone               → Managed, serverless, easy scaling
Weaviate               → Hybrid search (vector + keyword), multi-model
Qdrant                 → High performance, Rust-based, self-hostable
Chroma                 → Local development, prototyping
Milvus                 → Enterprise scale, GPU acceleration

// ❌ HALLUCINATION TRAP: Vector search is NOT keyword search
// "Apple CEO" might not find "Tim Cook runs Apple Inc."
// Use HYBRID search (vector + BM25 keyword) for production
```

---

## Streaming

```typescript
// Server-Sent Events for AI token streaming
app.get('/api/chat', async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  const stream = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: req.query.message as string }],
    stream: true,
  });

  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content;
    if (content) {
      res.write(`data: ${JSON.stringify({ content })}\n\n`);
    }
  }

  res.write('data: [DONE]\n\n');
  res.end();
});

// Client-side consumption
const eventSource = new EventSource(`/api/chat?message=${encodeURIComponent(msg)}`);
eventSource.onmessage = event => {
  if (event.data === '[DONE]') {
    eventSource.close();
    return;
  }
  const { content } = JSON.parse(event.data);
  appendToChat(content);
};
```

---

## Cost Optimization

```
1. Prompt caching        → Cache system prompts (OpenAI, Anthropic support this)
2. Output token limiting → Set max_tokens to prevent runaway responses
3. Tiered models         → Use cheap models for classification, expensive for reasoning
4. Batch processing      → Use batch APIs for offline processing (50% discount)
5. Chunked context       → Send only relevant chunks, not entire documents
6. Response streaming    → Stream to reduce TTFT (time to first token)
7. Structured output     → Shorter JSON responses vs verbose prose

// Cost estimation:
// GPT-4o: ~$2.50/1M input, ~$10/1M output
// GPT-4o-mini: ~$0.15/1M input, ~$0.60/1M output
// 1M tokens ≈ 750,000 words ≈ 3,000 pages
```

## 🚨 Edge-Case & Failure Mode Matrix

| Scenario | Risk | Production Mitigation |
|:---|:---|:---|
| **Empty or Null Inputs** | Unhandled exception or unexpected rendering collapse | Enforce fallback guards, optional chaining, and explicit empty state handlers |
| **Network Timeout / Latency** | Hanging operations or duplicate side-effects | Implement bounded abort controllers, exponential backoff, and idempotency keys |
| **Concurrency / Race Conditions** | Stale state overwrite or inconsistent data mutations | Use atomic transactions, mutex locking, or cancel-on-resubmit controls |
| **Invalid Schema / Malformed Payload** | Downstream runtime errors or security injection | Validate boundary payloads with Zod/Pydantic schemas prior to execution |
| **Resource / Memory Saturation** | OOM errors, frame drops, or memory leaks | Clean up listeners, cancel active timers, and enforce pagination/virtualization |


## 🤖 LLM-Specific Traps Table

| Anti-Pattern | What AI Commonly Does Wrong | What Is Actually Correct |
|:---|:---|:---|
| **Unchecked Payload Cast** | Casting request bodies to TypeScript types without runtime schema validation | Parse request payloads through Zod/Pydantic schemas before business logic |
| **Silent Error Swallowing** | Catching errors with empty catch blocks or logging without rethrowing | Propagate structured errors with status codes and contextual stack traces |
| **Unparameterized Query** | Concatenating user inputs into SQL/Prisma query strings | Always use parameterized bindings or type-safe ORM query builders |


## 🏛️ Tribunal Verification & Guardrails

**Active Reviewers:** `logic-reviewer` · `security-auditor` · `api-architect` · `resilience-reviewer`
**Slash Command:** `/review` or `/tribunal-full`

### 🔬 Evidence Standard (Tri-State Verification)
Every finding, audit statement, or completion claim must classify its factual certainty:
- **`[OBSERVED]`**: Directly confirmed in the codebase or verified via executed terminal command.
- **`[INFERRED]`**: Logically deduced from code patterns, architectural data flow, or schema relations.
- **`[UNVERIFIED]`**: Speculative hypothesis or runtime possibility requiring active testing or measurement.

### ✅ Pre-Flight Self-Audit Checklist
```
✅ Are all inputs and boundary payloads validated against schemas (Zod/Pydantic)?
✅ Are SQL and database queries parameterized with zero string concatenation?
✅ Are error boundaries and timeout/retry policies explicitly declared?
✅ Are authentication and object-level authorization (IDOR/BOLA) checked before business logic?
✅ Did I verify that imported dependencies exist in package manifests?
```

### 🛑 Verification-Before-Completion (VBC) Protocol
**CRITICAL:** You must follow a strict "evidence-based closeout" state machine.
- ❌ **Forbidden:** Declaring a task complete because the output "looks correct."
- ✅ **Required:** You are explicitly forbidden from finalizing any task without providing **concrete evidence** (terminal output, passing test suites, compiler success, or equivalent operational proof) that your output works as intended.