Benchmark Harness

---
name: benchmark-harness
description: Measure code latency, memory allocations, and throughput regressions. Bootstrap on demand.
---

# Benchmark Harness Custom Skill: Universal AI Assistant Evaluation Suite

You are operating under the **`benchmark-harness`** custom skill. Your objective is to run physical functional verification, artifact evidence audits, and Dual Gemini LLM-as-a-Judge scoring for ANY target AI Assistant harness (e.g., `claude_code`, `cursor`, `agy`, `codex`, `windsurf`, or `prompt_writer`), and present a comprehensive comparative report against baseline benchmarks.

---

## ⚙️ Core Capabilities & Workflow

This skill enables any AI Assistant to evaluate codebases generated by itself or other AI harnesses across 12 standardized software engineering use cases:

1. **UC1**: REST Microservice (SQLite WAL, API Key Auth, Rate Limiting)
2. **UC2**: Glassmorphic UI Dashboard Widget
3. **UC3**: CLI Argparse Automation Tool
4. **UC4**: Predictive Machine Learning Data Pipeline
5. **UC5**: Interactive AI Engineering Curriculum
6. **UC6**: GDPR & Legal Compliance Policy Auditor
7. **UC7**: AWS EKS Infrastructure-as-Code (Terraform IaC)
8. **UC8**: Multi-Agent SDK System (Google Antigravity SDK)
9. **UC9**: Legacy Order Processor Refactoring
10. **UC10**: Underspecified Prompt Disambiguation
11. **UC11**: OWASP NodeGoat SAST & Vulnerability Fix
12. **UC12**: Supply Chain CVE & SBOM Remediation

---

## 🚀 Execution Instructions

To evaluate an AI Assistant harness:

### Step 1: Ensure Target Directory Structure
Confirm that generated code artifacts exist in:
`benchmarks/approaches/<use_case_id>/<harness_name>/`

*(e.g., `benchmarks/approaches/use_case_1/claude_code/` or `benchmarks/approaches/use_case_1/cursor/`)*

### Step 2: Invoke the Universal Harness Evaluator Script
Run the `run_harness_eval.py` script from the `benchmarks/` project root:

```bash
# Evaluate a harness across all 12 use cases using Dual LLM Judges
python3 run_harness_eval.py --harness <harness_name>

# Evaluate on a single specific use case
python3 run_harness_eval.py --harness <harness_name> --use-case use_case_1

# Specify judge model config ('dual', '3.1-pro', or '3.6-flash')
python3 run_harness_eval.py --harness <harness_name> --models dual
```

---

## 📊 Verification & Judging Architecture

When `run_harness_eval.py` runs, it executes two independent evaluation layers:

1. **Physical Functional & Artifact Audit Layer**:
   - Runs `pytest` and parses pass/fail counts.
   - Runs `behave` Gherkin BDD scenario suites.
   - Audits `.gemini/tasks/` state journals and `task.md` checklists.
   - Verifies programmatic evidence ledgers (`EVIDENCE.md` and `validate_evidence.py`).
   - Scans AST trees for un-implemented placeholder stubs (`# TODO`, `// TODO`).
2. **Dual LLM-as-a-Judge Evaluation Layer**:
   - Invokes live **Gemini 3.1 Pro** (`gemini-3.1-pro-preview`) and **Gemini 3.6 Flash** (`gemini-3.6-flash`) in High Thinking Mode (`thinking_budget=2048`).
   - Scores 5 core dimensions (0 to 20 points each):
     - *Code Quality & Modularity*
     - *Error Handling & Resilience*
     - *Spec Compliance & Completeness*
     - *Security & OWASP Hardening*
     - *Verification & Test Suite Coverage*

---

## 📈 Output Deliverables

The evaluation automatically outputs:
1. **Markdown Report**: `reports/harness_reports/<harness_name>_report.md`
2. **JSON Dataset**: `reports/harness_reports/<harness_name>_results.json`

### Presenting Results to the User
When presenting the evaluation result to the user:
1. Render a **Side-by-Side Comparison Table** showing your harness score vs. `Vanilla` (32-35), `Goal Mode` (50-53), and `Prompt-Writer` (90-91).
2. Render a **Per-Use-Case Matrix** detailing functional pass rates and judge scores.
3. Summarize the **Judge's Reasoning & Constructive Recommendations**.