Testing Conformance Harnesses

---
name: testing-conformance-harnesses
description: >-
  Build conformance test harnesses that verify implementations against specifications.
  Use when: porting libraries across languages, implementing RFCs/specs, building
  database engines, validating protocol compliance, cross-platform compatibility,
  API contract testing, golden file testing, round-trip validation, compliance
  matrices, differential testing against reference implementations.
metadata:
  filePattern:
    - "**/conformance*"
    - "**/golden*"
    - "**/fixtures/**"
    - "**/reference*"
  bashPattern:
    - "\\b(conformance|compliance|golden|differential|reference.impl|wire.compat)\\b"
  priority: 60
---

# Conformance Test Harnesses

> **The One Rule:** "Specifications aren't suggestions, they're contracts."
> A conformance harness mechanically verifies every MUST/SHOULD clause.
> If it's not tested, it's not conformant.

## The Loop (Mandatory)

```
1. IDENTIFY    → What is the specification? (RFC, API spec, reference impl, formal grammar)
2. EXTRACT     → Enumerate every testable requirement (MUST > SHOULD > MAY)
3. FIXTURE     → Generate reference outputs (run reference impl → golden files)
4. HARNESS     → Build infrastructure: fixture loader, comparator, verdict engine
5. COVER       → Write tests: one per requirement, table-driven, tagged by level
6. DIVERGE     → Document every INTENTIONAL deviation in DISCREPANCIES.md
7. MATRIX      → Generate compliance report: features × status × platform
8. MAINTAIN    → Regenerate fixtures when reference impl updates; diff review
```

## Coverage Accounting Matrix (Mandatory)

Before claiming conformance, prove it:

| Spec Section | MUST Clauses | SHOULD Clauses | Tested | Passing | Divergent | Score |
|-------------|:-----------:|:--------------:|:------:|:-------:|:---------:|-------|
| *Section N* | count | count | count | count | count | Pass/(MUST+SHOULD) |

**Rule:** Score < 0.95 for MUST clauses = NOT conformant. Ship with known gaps
documented, never with unknown gaps.

---

## Decision Tree: Which Conformance Pattern?

```
What is the specification source?
│
├─ Reference implementation exists (Go, Python, C)
│  └─ DIFFERENTIAL TESTING (Pattern 1)
│     Run both implementations, compare outputs byte-for-byte
│     Examples: charmed_rust (Go→Rust), mcp_agent_mail_rust (Python→Rust)
│
├─ Formal spec exists (RFC, ISO, W3C)
│  └─ SPEC-DERIVED TESTS (Pattern 4)
│     One test per MUST/SHOULD clause, tagged by requirement level
│     Examples: JSON RFC 7159, HTTP/2 RFC 7540, SQL spec
│
├─ Serialization format
│  └─ ROUND-TRIP + GOLDEN FILES (Patterns 2 + 3)
│     Fixtures from reference impl + serialize→deserialize identity
│     Examples: protobuf, MessagePack, CBOR, database page format
│
├─ Network protocol
│  └─ PROCESS-BASED CONFORMANCE (Pattern 6)
│     External test runner drives your server/client
│     Examples: Connect conformance suite, WPT, test262
│
└─ API contract (OpenAPI, GraphQL)
   └─ CONTRACT TESTING (Pattern 5)
      Consumer-driven contracts, provider verification
      Examples: Pact, Dredd, Hurl
```

---

## The Six Patterns

### Pattern 1: Differential Testing (Reference Implementation)

**Use when:** A canonical implementation exists. This is the gold standard.

**Architecture (from charmed_rust, mcp_agent_mail_rust):**

```
tests/conformance/
├── src/
│   ├── harness/
│   │   ├── mod.rs          # Entry point
│   │   ├── traits.rs       # ConformanceTest trait (see below)
│   │   ├── runner.rs       # Collects + executes all tests
│   │   ├── fixtures.rs     # Loads golden files from reference impl
│   │   ├── comparison.rs   # Byte-level, structural, fuzzy comparison
│   │   ├── context.rs      # Test context: paths, config, temp dirs
│   │   └── logging.rs      # Structured JSON-line results
│   └── bin/
│       ├── run_conformance.rs    # `cargo run --bin run_conformance`
│       └── generate_report.rs    # Markdown compliance matrix
├── fixtures/
│   └── go_outputs/          # Generated by: go run ./cmd/gen-fixtures
│       └── lipgloss/
│           ├── border_rounded.golden
│           └── style_padding.golden
├── DISCREPANCIES.md          # Every intentional divergence
└── COVERAGE.md               # What's tested vs what's not
```

**The ConformanceTest Trait:**

```rust
pub trait ConformanceTest: Send + Sync {
    fn name(&self) -> &str;
    fn category(&self) -> TestCategory;
    fn requirement_level(&self) -> RequirementLevel;  // MUST, SHOULD, MAY
    fn run(&self, ctx: &TestContext) -> TestResult;
}

#[derive(Debug, Serialize)]
pub enum TestCategory { Unit, Integration, EdgeCase, Performance }

#[derive(Debug, Serialize)]
pub enum RequirementLevel { Must, Should, May }

#[derive(Debug, Serialize)]
#[serde(tag = "status")]
pub enum TestResult {
    Pass,
    Fail { reason: String },
    Skipped { reason: String },
    ExpectedFailure { reason: String },  // Known divergence (XFAIL)
}
```

**Fixture-driven differential test:**

```rust
#[test]
fn conformance_lipgloss_border_rounded() {
    let fixture = load_fixture("go_outputs/lipgloss/border_rounded.golden");
    let actual = Style::new()
        .border(Border::Rounded)
        .padding(1, 2)
        .render("Hello, World!");

    assert_eq!(actual, fixture.expected,
        "Rust rendering diverges from Go reference\n\
         Go output:   {:?}\n\
         Rust output: {:?}\n\
         Fixture:     {}",
        fixture.expected, actual, fixture.path.display());
}
```

### Pattern 2: Golden File Testing

**Use when:** Output is complex, correct once verified, then frozen.

```rust
fn assert_golden(test_name: &str, actual: &str) {
    let golden_path = Path::new("tests/golden")
        .join(format!("{test_name}.golden"));

    if std::env::var("UPDATE_GOLDENS").is_ok() {
        fs::create_dir_all(golden_path.parent().unwrap()).unwrap();
        fs::write(&golden_path, actual).unwrap();
        eprintln!("UPDATED golden: {}", golden_path.display());
        return;
    }

    let expected = fs::read_to_string(&golden_path)
        .unwrap_or_else(|_| panic!(
            "Golden file not found: {}\n\
             Run with UPDATE_GOLDENS=1 to create it",
            golden_path.display()
        ));

    if actual != expected {
        let actual_path = golden_path.with_extension("actual");
        fs::write(&actual_path, actual).unwrap();
        panic!(
            "GOLDEN MISMATCH: {}\n\
             diff {} {}",
            test_name,
            golden_path.display(),
            actual_path.display(),
        );
    }
}
```

**Workflow:**
```bash
# First run: create golden files
UPDATE_GOLDENS=1 cargo test

# Subsequent runs: compare
cargo test

# After intentional changes:
diff tests/golden/report.golden tests/golden/report.actual
UPDATE_GOLDENS=1 cargo test
git diff tests/golden/  # Review every change before committing
```

### Pattern 3: Round-Trip Conformance

**Use when:** Data must survive a serialize→deserialize cycle perfectly,
AND must interoperate with a reference implementation.

```rust
/// Cross-implementation round-trip:
/// 1. Reference impl produces fixture
/// 2. Our impl parses it (must succeed)
/// 3. Our impl re-serializes it
/// 4. Reference impl parses our output (must succeed and match)
fn conformance_cross_impl_roundtrip(fixtures_dir: &Path) {
    for fixture_path in glob(fixtures_dir, "*.bin") {
        let reference_bytes = fs::read(&fixture_path).unwrap();

        // Step 1: We must be able to parse reference output
        let parsed = our_parser::parse(&reference_bytes)
            .unwrap_or_else(|e| panic!(
                "Cannot parse reference fixture {}: {e}",
                fixture_path.display()));

        // Step 2: We re-serialize
        let our_bytes = our_serializer::serialize(&parsed);

        // Step 3: Reference must be able to parse our output
        let reparsed = reference_parser::parse(&our_bytes)
            .unwrap_or_else(|e| panic!(
                "Reference cannot parse our output for {}: {e}",
                fixture_path.display()));

        // Step 4: Parsed values must match
        assert_eq!(parsed, reparsed,
            "Cross-impl round-trip diverged for {}",
            fixture_path.display());
    }
}
```

### Pattern 4: Spec-Derived Test Matrix

**Use when:** Implementing an RFC or formal specification.

```rust
struct ConformanceCase {
    id: &'static str,           // "RFC7159-2.1"
    section: &'static str,      // "2"
    level: RequirementLevel,    // Must, Should, May
    description: &'static str,
    input: &'static str,
    expected: Result<Value, ()>,
}

const RFC7159_CASES: &[ConformanceCase] = &[
    // Section 2: JSON Grammar
    ConformanceCase {
        id: "RFC7159-2.1",
        section: "2",
        level: RequirementLevel::Must,
        description: "A JSON text is a serialized value",
        input: "42",
        expected: Ok(Value::Number(42)),
    },
    ConformanceCase {
        id: "RFC7159-7.1",
        section: "7",
        level: RequirementLevel::Must,
        description: "Unicode escape sequences \\uXXXX",
        input: r#""\u0041""#,
        expected: Ok(Value::String("A".into())),
    },
    // ... one per MUST/SHOULD clause
];

#[test]
fn rfc7159_full_conformance() {
    let mut pass = 0;
    let mut fail = 0;
    let mut xfail = 0; // Expected failures (known divergences)

    for case in RFC7159_CASES {
        let result = our_parser::parse(case.input);
        let verdict = match (&result, &case.expected) {
            (Ok(a), Ok(b)) if a == b => { pass += 1; "PASS" }
            (Err(_), Err(())) => { pass += 1; "PASS" }
            _ => {
                if is_known_divergence(case.id) {
                    xfail += 1;
                    "XFAIL"
                } else {
                    fail += 1;
                    eprintln!("FAIL {}: {}\n  expected: {:?}\n  actual: {:?}",
                        case.id, case.description, case.expected, result);
                    "FAIL"
                }
            }
        };
        // Structured JSON-line output for CI parsing
        eprintln!("{{\"id\":\"{}\",\"verdict\":\"{verdict}\",\"level\":\"{:?}\"}}",
            case.id, case.level);
    }

    let total = pass + fail + xfail;
    eprintln!("\nRFC 7159: {pass}/{total} pass, {fail} fail, {xfail} expected-fail");
    assert_eq!(fail, 0, "{fail} conformance tests failed");
}
```

### Pattern 5: Contract Testing (API Conformance)

**Use when:** Testing API compatibility between services.

```typescript
// Consumer-driven contract: client defines expectations
// Provider must satisfy ALL consumer contracts

// Consumer side (tests what we NEED from the API)
describe("User API Contract", () => {
  it("GET /users/:id returns user with email", async () => {
    const user = await api.getUser("user-123");
    expect(user).toHaveProperty("id");
    expect(user).toHaveProperty("email");
    expect(typeof user.email).toBe("string");
    // Contract: we don't care about other fields — only what we consume
  });
});

// Provider side (verifies all consumer contracts are satisfied)
// Run against actual API implementation
```

```bash
# Hurl: HTTP-level conformance testing
# tests/conformance/api/users.hurl
GET http://localhost:3000/api/v1/users/me
Authorization: Bearer {{token}}
HTTP 200
[Asserts]
jsonpath "$.id" exists
jsonpath "$.email" isString
jsonpath "$.subscriptionStatus" matches /^(none|active|past_due|cancelled)$/
```

### Pattern 6: Process-Based Conformance (External Runner)

**Use when:** A standard conformance runner exists for your protocol.

```bash
# Connect conformance suite: tests gRPC/Connect protocol compliance
connectconformance --mode server \
  --config conformance-config.yaml \
  -- ./our-server

# Web Platform Tests: browser conformance
wpt run --channel dev --product our-browser

# test262: JavaScript engine conformance
test262-harness --hostType our-engine --hostPath ./our-engine
```

---

## DISCREPANCIES.md (Mandatory)

Every conformance harness accumulates intentional divergences. Document them ALL.

```markdown
# Known Conformance Divergences

## DISC-001: Unicode width tables
- **Reference:** Uses Unicode 13.0 width tables (go-runewidth v0.14)
- **Our impl:** Uses Unicode 15.1 width tables (unicode-width v0.2)
- **Impact:** Some CJK chars have different widths → alignment differs
- **Resolution:** ACCEPTED — newer Unicode tables are more correct
- **Tests affected:** lipgloss/cjk_alignment_*
- **Review date:** 2026-03-15

## DISC-002: Error message format
- **Reference:** Returns "invalid input at byte 42"
- **Our impl:** Returns "parse error: unexpected byte 0x2A at offset 42"
- **Impact:** Error strings differ (semantics identical)
- **Resolution:** ACCEPTED — we test error categories, not messages
- **Tests affected:** parser/error_*
```

**Rules for DISCREPANCIES.md:**
1. Every divergence gets a sequential ID (DISC-NNN)
2. Must state whether ACCEPTED, INVESTIGATING, or WILL-FIX
3. Must list affected test cases
4. Must include review date (divergences can become stale)
5. Tests for accepted divergences use XFAIL, not SKIP

---

## Fixture Provenance (Non-Negotiable)

Every fixture must record how it was generated:

```
tests/conformance/fixtures/
├── PROVENANCE.md              # How fixtures were generated
├── go_outputs/
│   ├── generated_with: go1.22.1
│   ├── command: go run ./cmd/gen-fixtures > fixtures/go_outputs/
│   └── git_ref: abc123 (tag: v0.15.2)
└── python_reference.json
    ├── generated_with: python3.12 + mcp-agent-mail 0.9.1
    └── command: python -m mcp_agent_mail.conformance.generate > python_reference.json
```

**Why:** When fixtures are regenerated 6 months later and results change,
you need to know what version generated the originals to diagnose whether
the change is a bug or a new feature.

---

## Compliance Report Generator

```rust
fn generate_compliance_report(results: &[TestResult]) -> String {
    let mut by_section: BTreeMap<&str, SectionStats> = BTreeMap::new();

    for result in results {
        let section = by_section.entry(result.section).or_default();
        match result.level {
            Must => section.must_total += 1,
            Should => section.should_total += 1,
            May => section.may_total += 1,
        }
        if result.verdict == Pass { section.passing += 1; }
        if result.verdict == XFail { section.xfail += 1; }
    }

    // Output: Markdown table
    // | Section | MUST (pass/total) | SHOULD (pass/total) | Score |
    // |---------|-------------------|---------------------|-------|
    // | §2      | 15/15             | 8/10                | 95.8% |
}
```

---

## Anti-Patterns (Hard Constraints)

| ✗ Never | Why | Fix |
|---------|-----|-----|
| Test implementation details, not spec behavior | Brittle, breaks on refactors | Test observable behavior only |
| No DISCREPANCIES.md | Intentional divergences look like bugs to next developer | Document EVERY known deviation |
| Golden files without `UPDATE_GOLDENS` workflow | Tedious to update → people skip updates | Add update mechanism + diff review |
| Incomplete COVERAGE.md | False confidence in compliance | Track what ISN'T tested |
| Fixtures without provenance | Can't reproduce or upgrade | Record generator version + command |
| SKIP instead of XFAIL for known divergences | Skipped tests are invisible in reports | XFAIL documents AND tracks |
| Test only happy paths | Error handling divergences are the most dangerous | Test invalid inputs too |
| Regenerate fixtures without diff review | New fixture bugs look like passing tests | Always `git diff fixtures/` before commit |

---

## Checklist (Before Claiming Conformance)

- [ ] Specification source identified and version pinned
- [ ] Coverage matrix built: every MUST/SHOULD clause enumerated
- [ ] MUST clause coverage ≥ 95% (100% target)
- [ ] Fixtures generated from reference impl with recorded provenance
- [ ] DISCREPANCIES.md documents every intentional divergence
- [ ] XFAIL (not SKIP) for accepted divergences
- [ ] Compliance report generated automatically
- [ ] Round-trip tests for all serializable types
- [ ] Error cases tested (not just happy paths)
- [ ] CI regenerates report on every PR
- [ ] Fixture update workflow documented and tested

---

## References

| Need | Reference |
|------|-----------|
| Harness architecture deep-dive | [HARNESS-ARCHITECTURE.md](references/HARNESS-ARCHITECTURE.md) |
| Fixture management patterns | [FIXTURE-PATTERNS.md](references/FIXTURE-PATTERNS.md) |
| Cross-language porting guide | [CROSS-LANGUAGE.md](references/CROSS-LANGUAGE.md) |
| Real-world examples from our projects | [OUR-PROJECTS.md](references/OUR-PROJECTS.md) |

## Relationship to Other Testing Skills

| Technique | Use INSTEAD when | Use TOGETHER when |
|-----------|-----------------|-------------------|
| /testing-metamorphic | No spec exists (oracle problem) | MRs fill gaps where spec is ambiguous |
| /testing-fuzzing | Finding crashes, not compliance | Fuzz-generated inputs feed conformance checks |
| /extreme-software-optimization | Performance, not correctness | Conformance suite is the regression gate for optimizations |
| /porting-to-rust | Need the full porting methodology | Conformance harness is PART of the porting workflow |