Prompt Template Manager

---
license: Apache-2.0
name: prompt-template-manager
description: "Version-control, parameterize, and A/B test LLM prompt templates with Git-native workflows. Activate on: prompt versioning, prompt templates, A/B test prompts, manage prompts, prompt registry. NOT for: writing prompts from scratch (prompt-engineer), fine-tuning data (fine-tuning-dataset-curator)."
allowed-tools: Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*)
category: AI & Machine Learning
tags:
  - prompt-management
  - versioning
  - a-b-testing
  - templates
  - llm-ops
pairs-with:
  - skill: prompt-engineer
    reason: Prompt engineer writes content; this skill manages lifecycle
  - skill: llm-evaluation-harness
    reason: A/B tests need automated evaluation to pick winners
  - skill: llm-cost-optimizer
    reason: Template compression and model routing reduce per-prompt cost
---

# Prompt Template Manager

Version-control, parameterize, and A/B test LLM prompt templates using Git-native workflows and structured registries.

## Activation Triggers

**Activate on**: "prompt versioning", "prompt template", "A/B test prompts", "prompt registry", "manage prompt variants", "prompt as code", "parameterized prompts", "prompt lifecycle"

**NOT for**: Writing or optimizing individual prompts (prompt-engineer), fine-tuning dataset preparation (fine-tuning-dataset-curator), or LLM application architecture (ai-engineer)

## Quick Start

1. **Structure prompts as files** — Store each prompt template in `prompts/{domain}/{name}.yaml` with metadata, variables, and versioned content.
2. **Parameterize with variables** — Use `{{variable}}` placeholders for dynamic content; separate static instruction from dynamic context.
3. **Version with Git** — Each prompt change is a commit. Tags mark production versions. Branches for experiments.
4. **A/B test** — Deploy multiple variants simultaneously, route traffic by percentage, collect metrics per variant.
5. **Promote winners** — Evaluate via automated harness or human review, promote winning variant to production tag.

## Core Capabilities

| Domain | Technologies | Notes |
|--------|-------------|-------|
| **Template Format** | YAML + Jinja2, Handlebars, Mustache | YAML frontmatter for metadata, body for template |
| **Version Control** | Git tags, branches, semantic versioning | `v1.2.0` tags for production, branches for experiments |
| **A/B Testing** | Feature flags (LaunchDarkly, Unleash, custom) | Percentage-based routing with metrics collection |
| **Registry** | File-based, PostgreSQL, Redis | Central lookup for template resolution at runtime |
| **Evaluation** | LLM-as-judge, human eval, RAGAS | Automated comparison of variant outputs |
| **Rendering** | Jinja2 (Python), Handlebars (JS), custom | Variable interpolation with type validation |

## Architecture Patterns

### Pattern 1: File-Based Prompt Registry

```
prompts/
├── customer-support/
│   ├── ticket-classifier.yaml      # Active production template
│   ├── ticket-classifier.v2.yaml   # Experiment variant
│   └── response-generator.yaml
├── content/
│   ├── blog-outline.yaml
│   └── social-post.yaml
└── _shared/
    ├── system-safety.yaml          # Reusable system prompt fragments
    └── output-format-json.yaml
```

```yaml
# prompts/customer-support/ticket-classifier.yaml
name: ticket-classifier
version: "1.3.0"
model: claude-sonnet-4-20250514
temperature: 0
description: Classify support tickets into categories
variables:
  - name: ticket_text
    type: string
    required: true
  - name: categories
    type: list
    default: [billing, technical, account, other]
includes:
  - _shared/output-format-json.yaml

template: |
  You are a support ticket classifier.

  Classify the following ticket into exactly one category.
  Categories: {{categories | join(", ")}}

  Ticket:
  {{ticket_text}}

  {{> output-format-json}}

tests:
  - input: { ticket_text: "I can't log in to my account" }
    expected_category: "account"
  - input: { ticket_text: "You charged me twice" }
    expected_category: "billing"
```

### Pattern 2: A/B Testing Pipeline

```
Request ──→ [Router] ──→ Variant A (control, 80%) ──→ [LLM] ──→ Response + Log
                │                                                      │
                └──→ Variant B (experiment, 20%) ──→ [LLM] ──→ Response + Log
                                                                       │
                                                                       ▼
                                                              [Metrics Store]
                                                              - latency
                                                              - token count
                                                              - quality score
                                                              - user feedback
                                                                       │
                                                                       ▼
                                                              [Evaluation]
                                                              Winner → promote to 100%
```

```python
# A/B routing with metrics
import random, time

class PromptRouter:
    def __init__(self, registry, metrics):
        self.registry = registry
        self.metrics = metrics

    def resolve(self, template_name: str, user_id: str) -> dict:
        variants = self.registry.get_variants(template_name)
        # Deterministic assignment by user_id for consistency
        bucket = hash(f"{user_id}:{template_name}") % 100
        for variant in variants:
            if bucket < variant["traffic_pct"]:
                self.metrics.log("variant_assigned", {
                    "template": template_name,
                    "variant": variant["version"],
                    "user_id": user_id
                })
                return variant
            bucket -= variant["traffic_pct"]
        return variants[0]  # Default to control
```

### Pattern 3: Prompt Composition with Includes

```
System Prompt = [safety-preamble] + [role-definition] + [output-format]
User Prompt   = [context-injection] + [user-query] + [constraints]

Compose from reusable fragments:
  _shared/safety-preamble.yaml  ──→ "You must not generate harmful content..."
  _shared/json-output.yaml     ──→ "Respond with valid JSON matching this schema..."
  domain/role.yaml              ──→ "You are an expert in {{domain}}..."

Final prompt = render(compose([safety, role, output]), variables)
```

## Anti-Patterns

1. **Prompts hardcoded in application code** — Changing a prompt requires a code deploy. Store prompts externally with hot-reload capability.
2. **No version tracking** — "Which prompt was running when quality dropped last Tuesday?" Without versions tied to timestamps, this is unanswerable.
3. **A/B testing without statistical significance** — Declaring a winner after 50 requests is noise. Require minimum sample sizes (typically 200+ per variant) and p-value thresholds.
4. **Monolithic mega-prompts** — A 3,000-token prompt that handles every case is fragile. Compose from reusable fragments so changes are isolated.
5. **No rollback plan** — Every prompt version must be instantly rollbackable by re-tagging the previous version as production.

## Quality Checklist

- [ ] Prompts stored as files with YAML metadata (not hardcoded in application code)
- [ ] Variables parameterized with types and validation (required, default, enum)
- [ ] Each prompt has inline test cases for regression testing
- [ ] Git tags mark production versions (semantic versioning: major.minor.patch)
- [ ] A/B test routing is deterministic per user (no flickering between variants)
- [ ] Minimum sample size enforced before declaring A/B test winners
- [ ] Metrics collected per variant: latency, tokens, quality score, user feedback
- [ ] Reusable fragments extracted to `_shared/` for composition
- [ ] Rollback possible in under 1 minute (re-tag or config flip)
- [ ] Prompt rendering tested: all variables substituted, no raw `{{placeholders}}` in output