Harness Engineering

---
name: harness-engineering
description: Design runtime infrastructure around AI agents — permissions, tools, feedback loops, observability. Use when deploying agents to production or designing multi-agent systems.
version: "1.2"
updated: "2026-05-18"
---

# Harness Engineering

Design the system *around* AI agents for reliable, safe production use.

## Agent Runtime Model

Treat the agent harness as the kernel around an LLM OS:

| Kernel concern | Agent harness responsibility |
|---|---|
| Memory management | Curate context, summarize bulky outputs, persist state to wiki/logs. |
| Syscall boundary | Expose tools with contracts, allowlists, deny rules, and retries. |
| Process isolation | Separate write scopes, sandboxes, credentials, and state per agent. |
| Scheduling | Decide sequential, parallel, or event-driven execution. |
| Interrupts | Stop, ask approval, rollback, or route to a safer action. |
| Observability | Log tool calls, decisions, outputs, costs, and verification evidence. |
| Garbage collection | Close idle agents, remove stale tasks, compact context, and record risks. |

## Usage Template

**Prompt**
```text
Use harness-engineering for this agent workflow. Design permissions, tools, feedback loops, observability, and failure handling.
```

**Use Case**
- Moving an agent workflow from ad hoc prompting toward a reliable runtime architecture.

**Expected Result**
- The agent produces a harness design with permission tiers, tool boundaries, logs, evals, and recovery paths.

**Output Example**
- A runtime spec with permission matrix, tool allowlist, approval gates, logs, evals, and incident response.

**Verification Case**
- The design names what the agent can do automatically, what needs approval, and what is denied.

**Verified Effect**
- An ad hoc agent workflow becomes a controlled runtime with explicit permissions, observability, and failure handling.

## When to Use

- Deploying agents to production
- Setting up permissions/guardrails/approval workflows
- Designing multi-agent systems
- Agent behaved unpredictably → needs better constraints
- Configuring auto-mode or permission tiers

---

## Three Domains

| Domain | Object | Maturity |
|--------|--------|----------|
| **Physical** | Wire harnesses (automotive/aerospace) | ⭐ Mature |
| **Software** | CI/CD pipelines (Harness.io) | ⭐ Mature |
| **Cognitive** ⭐ | AI Agents | 🌱 Emerging |

---

## Six Components

### 1. Context & Knowledge Layer
- Curated access to code, docs, schemas, logs
- Use `CLAUDE.md` for project-level context
- Use `context-manager` for token budgeting
- Never inject raw 10K+ token files
- Persist reusable outputs to wiki, docs, logs, or state files; chat history is not durable memory

### 2. Tooling & API Surface
**Three-Tier Permission Model:**

| Tier | Scope | Mechanism | Examples |
|:----:|-------|-----------|----------|
| **1** | Safe tools | Always allowed | Read, search, grep, glob |
| **2** | In-project | Auto-approve (git reviewable) | Write/edit in project dir |
| **3** | High-risk | Classifier/human approval | Shell, API calls, deletes |

**Tier 3 heuristic:**
```
1. Can destroy data irreversibly? → BLOCK
2. Accesses credentials? → BLOCK
3. Affects shared infrastructure? → BLOCK
4. Target inferred (not explicit)? → BLOCK
```

### 3. Architectural Constraints
- Linters, structural tests
- File/directory access boundaries
- Token budget: never exceed 80% context window

### 4. Feedback & Validation Loops
- Write-Test-Fix cycle
- **Generator + Evaluator pattern** (GAN-inspired)
- Deny-and-continue: try safer alternative on block
- Escalate to human after 3 consecutive blocks

### 5. Observability & Governance
- Log every tool call + result
- Metrics: success rate, revert rate, token usage
- Cost tracking per session
- Evidence ledger: verification command, source link, screenshot, test result, or diff for each completion claim

### 6. Maintenance
- Periodic agents: dead code, outdated docs, architectural drift
- Wiki lint for broken links/missing frontmatter
- Context garbage collection: summarize completed work, close idle processes, and record residual risk

---

## GAN-Inspired Multi-Agent Pattern

> "Tuning a standalone evaluator to be skeptical is far more tractable than making a generator critical of its own work."

```
Generator ←──sprint contract──→ Evaluator
    │                              │
    ▼                              ▼
 Produces output              Playwright/Test
```

**Implementation:**
1. **Planner** → 1-sentence prompt → full spec
2. **Generator** → builds in sprints
3. **Evaluator** → tests, files bugs
4. Feedback loop → 5-15 iterations

---

## Physical → Cognitive Analogies

| Physical | AI Agent | Implementation |
|----------|----------|----------------|
| Zonal Architecture | Permission scopes | `AGENTS.md` domain restrictions |
| IP68 Protection | Security isolation | Sandbox, token in vault |
| HiPot Test | Stress testing | Edge-case test suite |
| Continuity Test | Tool verification | `describe-tools` pre-flight |
| EMC/Interference | Multi-agent isolation | Separate sandbox per agent |

---

## Multi-Agent Patterns

| Pattern | Description | When to Use |
|---------|-------------|-------------|
| **Supervisor** | Orchestrator delegates to sub-agents | Complex tasks with sub-steps |
| **Peer Review** | Agents review each other | High-risk changes |
| **Competitive** | Multiple agents try same task | Creative work, optimization |
| **Pipeline** | Sequential chain | Document processing |
| **Zonal** | Domain-based routing | Enterprise systems |
| **Closed-Loop** | Output feedback shapes input | Continuous improvement |

---

## Permissioned Tool Design

Define every tool like a system call:

| Field | Required question |
|---|---|
| Purpose | What state can this tool read or change? |
| Inputs | Which arguments must be explicit, not inferred? |
| Bounds | Which paths, domains, records, or resources are allowed? |
| Failure mode | What is the safe fallback when it fails or is denied? |
| Evidence | What output proves it succeeded? |
| Audit | Where is the call logged? |

Prefer narrow tools with explicit inputs over broad shell/API access. If broad access is unavoidable, wrap it with approval gates and post-action verification.

---

## Closed-Loop System (Aladdin)

```
Data Lake → Risk Engine → Optimizer → Stress Test → OMS → Compliance → Feedback Loop
     ↑                                                              ↓
     └──────────────────── 反馈闭环 ────────────────────────────────┘
```

| Aladdin Module | Agent Harness |
|----------------|---------------|
| Data Lake | Session log, knowledge base |
| Risk Engine | Risk assessment, threat detection |
| Optimizer | Task decomposition, resource allocation |
| Stress Test | Edge-case testing, failure simulation |
| OMS | Order execution, tool calls |
| Compliance | Guardrails, policy enforcement |
| Feedback Loop | Results → learning → improvement |

**Key:** Closed loop is the moat — not any single component. MECE principle applies.

---

## Quality Gates

- [ ] Each agent has defined permission scope
- [ ] Three-tier permission model documented
- [ ] Write-Test-Fix feedback loop in place
- [ ] Tool calls logged and observable
- [ ] Token costs tracked per session
- [ ] High-risk actions require approval
- [ ] Multi-agent isolation prevents interference
- [ ] Generator and Evaluator separate for complex tasks
- [ ] Reusable outputs have durable write-back outside chat history
- [ ] Tool contracts include bounds, failure path, evidence, and audit trail