Llm Evaluation

---
version: 4.1.0-fractal
name: llm-evaluation
description: Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
---

# LLM Evaluation

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.

## Do not use this skill when

- The task is unrelated to llm evaluation
- You need a different domain or tool outside this scope

## Instructions

- Clarify goals, constraints, and required inputs.
- Apply relevant best practices and validate outcomes.
- Provide actionable steps and verification.
- If detailed examples are required, open `resources/implementation-playbook.md`.

## Use this skill when

- Measuring LLM application performance systematically
- Comparing different models or prompts
- Detecting performance regressions before deployment
- Validating improvements from prompt changes
- Building confidence in production systems
- Establishing baselines and tracking progress over time
- Debugging unexpected model behavior

## Core Evaluation Types

## 🧠 Knowledge Modules (Fractal Skills)

### 1. [1. Automated Metrics](./sub-skills/1-automated-metrics.md)
### 2. [2. Human Evaluation](./sub-skills/2-human-evaluation.md)
### 3. [3. LLM-as-Judge](./sub-skills/3-llm-as-judge.md)
### 4. [BLEU Score](./sub-skills/bleu-score.md)
### 5. [ROUGE Score](./sub-skills/rouge-score.md)
### 6. [BERTScore](./sub-skills/bertscore.md)
### 7. [Custom Metrics](./sub-skills/custom-metrics.md)
### 8. [Single Output Evaluation](./sub-skills/single-output-evaluation.md)
### 9. [Pairwise Comparison](./sub-skills/pairwise-comparison.md)
### 10. [Annotation Guidelines](./sub-skills/annotation-guidelines.md)
### 11. [Inter-Rater Agreement](./sub-skills/inter-rater-agreement.md)
### 12. [Statistical Testing Framework](./sub-skills/statistical-testing-framework.md)
### 13. [Regression Detection](./sub-skills/regression-detection.md)
### 14. [Running Benchmarks](./sub-skills/running-benchmarks.md)