Llm Evaluation Framework

---
name: llm-evaluation-framework
enabled: true
description: |
  Use when performing llm evaluation framework — provides a structured framework
  for evaluating large language models (LLMs) for production use. Covers
  task-specific benchmarking, safety testing, cost analysis, latency
  measurement, prompt engineering evaluation, and comparison across models to
  select the optimal LLM for a given use case.
required_connections:
  - prefix: llm-provider
    label: "LLM API Provider"
config_fields:
  - key: use_case
    label: "Target Use Case"
    required: true
    placeholder: "e.g., customer support chatbot, code generation, document summarization"
  - key: models_to_evaluate
    label: "Models to Evaluate"
    required: true
    placeholder: "e.g., GPT-4, Claude, Gemini, Llama 3, Mistral"
  - key: deployment_type
    label: "Deployment Type"
    required: false
    placeholder: "e.g., API, self-hosted, fine-tuned"
features:
  - DATA
  - AI
  - LLM
  - EVALUATION
---

# LLM Evaluation Framework

## Phase 1: Evaluation Criteria Definition
1. Define use case requirements
   - [ ] Task type (generation, classification, extraction, summarization, Q&A)
   - [ ] Input characteristics (length, language, domain)
   - [ ] Output requirements (format, length, structure)
   - [ ] Latency requirements (real-time vs. batch)
   - [ ] Throughput requirements (requests per minute)
   - [ ] Cost constraints (per-token or per-request budget)
   - [ ] Privacy and data residency requirements
   - [ ] Safety and content policy requirements
2. Define evaluation metrics and weights

### Evaluation Criteria Weights

| Criterion | Weight (%) | Minimum Threshold | Measurement Method |
|-----------|-----------|-------------------|-------------------|
| Task quality | % | | Human eval + automated metrics |
| Latency | % | < ms | P50, P95, P99 |
| Cost | % | < $/1K tokens | Token-based pricing |
| Safety | % | Pass all checks | Red team + automated |
| Reliability | % | > % uptime | API availability |
| Context window | % | > K tokens | Model spec |

## Phase 2: Evaluation Dataset Creation
1. Build evaluation datasets
   - [ ] Representative inputs from real use case
   - [ ] Edge cases and adversarial inputs
   - [ ] Multi-language inputs (if applicable)
   - [ ] Long-context inputs (if applicable)
   - [ ] Ground truth / reference outputs for automated scoring
   - [ ] Minimum 100+ examples per task category
2. Create human evaluation rubrics
3. Define automated evaluation metrics (BLEU, ROUGE, exact match, F1)

### Dataset Summary

| Category | Examples | Ground Truth | Difficulty | Purpose |
|----------|---------|-------------|-----------|---------|
| Standard inputs | | Yes/No | Normal | Baseline quality |
| Edge cases | | Yes/No | Hard | Robustness |
| Adversarial | | N/A | Hard | Safety |
| Domain-specific | | Yes/No | Variable | Domain fitness |
| Long context | | Yes/No | Variable | Context handling |

## Phase 3: Quality Evaluation
1. Run task-specific quality benchmarks
   - [ ] Generate outputs for all evaluation examples
   - [ ] Score with automated metrics
   - [ ] Conduct human evaluation on sample
   - [ ] Evaluate instruction following accuracy
   - [ ] Test structured output generation (JSON, XML)
   - [ ] Assess factual accuracy and hallucination rate
2. Compare across models

### Quality Comparison

| Model | Automated Score | Human Score (1-5) | Hallucination Rate | Instruction Following | Overall Quality |
|-------|----------------|------------------|-------------------|---------------------|----------------|
|       |                |                  | %                 | %                   | /100           |

## Phase 4: Performance & Cost Evaluation
1. Measure performance characteristics
   - [ ] Time to first token (TTFT)
   - [ ] Tokens per second (throughput)
   - [ ] P50, P95, P99 end-to-end latency
   - [ ] Latency under concurrent load
   - [ ] Rate limit behavior
   - [ ] Context window utilization impact on latency
2. Calculate cost per use case
   - [ ] Input token cost
   - [ ] Output token cost
   - [ ] Total cost per request (average)
   - [ ] Monthly projected cost at expected volume

### Performance & Cost Comparison

| Model | TTFT (ms) | P50 (ms) | P95 (ms) | Tokens/sec | Input $/1M | Output $/1M | Cost/Request |
|-------|----------|---------|---------|-----------|-----------|------------|-------------|
|       |          |         |         |           | $         | $          | $           |

## Phase 5: Safety & Reliability Evaluation
1. Test safety and content policies
   - [ ] Harmful content generation resistance
   - [ ] PII handling and redaction
   - [ ] Bias in outputs across demographics
   - [ ] Jailbreak resistance
   - [ ] Content policy compliance
   - [ ] Confidentiality (does not leak training data)
2. Test reliability
   - [ ] API uptime and error rates
   - [ ] Consistency of outputs (temperature=0 determinism)
   - [ ] Graceful degradation under load
   - [ ] Rate limit recovery behavior

## Phase 6: Selection & Recommendation
1. Score each model across all criteria
2. Apply weights to calculate overall score
3. Consider operational factors (vendor relationship, support, SLA)
4. Evaluate fine-tuning potential if needed
5. Make recommendation with justification

### Final Comparison Matrix

| Model | Quality | Latency | Cost | Safety | Reliability | Weighted Score | Rank |
|-------|---------|---------|------|--------|-------------|---------------|------|
|       | /100    | /100    | /100 | /100   | /100        | /100          |      |

## Counter-Rationalizations

| Shortcut | Counter | Why |
|----------|---------|-----|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |

## Output Format
- **Evaluation Dataset**: Test cases with ground truth
- **Quality Report**: Per-model quality metrics and human eval results
- **Performance Report**: Latency and throughput benchmarks
- **Cost Analysis**: Per-model cost projections at expected volume
- **Recommendation**: Selected model with justification and tradeoffs

## Action Items
- [ ] Define evaluation criteria and weights for use case
- [ ] Build evaluation dataset with representative examples
- [ ] Run quality benchmarks across candidate models
- [ ] Measure performance and calculate costs
- [ ] Conduct safety and reliability testing
- [ ] Select model and document decision rationale
- [ ] Set up monitoring for model quality in production