Building Gold Corpus

---
name: building-gold-corpus
description: "Scaffold a synthetic gold-standard annotation project for evaluating OpenMed NER and de-identification models — label schema, annotation guidelines, BRAT or Label Studio config, and disjoint train/dev/test splits. Use when the user wants to create eval fixtures, set up annotation, define a label set, write guidelines, configure an annotation tool, or build a held-out gold set for the OpenMed eval harness. Trigger on \"gold corpus\", \"annotation project\", \"label schema\", \"annotation guidelines\", \"BRAT\", \"Label Studio\", \"train dev test split\", or \"build eval fixtures\" for OpenMed. Committed gold must be synthetic; licensed (i2b2/n2c2/MIMIC) data is eval-only and never committed."
license: Apache-2.0
metadata:
  project: OpenMed
  category: evaluation-quality
  pairs: adjacent
  version: "1.0"
---

# Building a Gold Corpus

You can't evaluate what you can't measure against. This skill scaffolds a
gold-standard annotation project whose output drops straight into the OpenMed
eval harness as fixtures. The hard rule: **anything committed to the repo is
synthetic**. Licensed clinical corpora (i2b2, n2c2, MIMIC) are DUA-gated — use
them at eval time from the user's own copy, never check them in.

## When to use this skill

- You need eval fixtures for `benchmarking-clinical-ner` or
  `evaluating-with-leakage-gates` and have none.
- You're standing up an annotation effort: schema, guidelines, tool config.
- You need disciplined train/dev/test splits with no leakage between them.
- You want a small **synthetic** golden set you can commit and gate on in CI.

## The OpenMed fixture shape (your target output)

Annotations must serialize to character-offset spans the harness understands:

```json
{
  "fixtures": [
    {
      "id": "synthetic-0001",
      "language": "en",
      "text": "Ms. Jane Roe (MRN 0000000) seen 2099-01-02 for type 2 diabetes.",
      "gold_spans": [
        {"start": 4,  "end": 12, "label": "PERSON"},
        {"start": 18, "end": 25, "label": "ID_NUM"},
        {"start": 32, "end": 42, "label": "DATE"},
        {"start": 47, "end": 62, "label": "DISEASE"}
      ]
    }
  ]
}
```

`openmed.eval.harness.load_fixtures` accepts a top-level list or a `{"fixtures":
[...]}` mapping. Offsets are character indices into `text`; labels are
OpenMed-canonical.

## Quick start — scaffold the project

```
eval/
  gold/
    guidelines.md            # annotation manual + edge-case decisions
    label_schema.json        # canonical labels + definitions + examples
    synthetic/               # COMMITTED synthetic fixtures (CI-gateable)
      train.json
      dev.json
      test.json
  external/                  # GITIGNORED: licensed DUA corpora, eval-only
    .gitignore               # *  (never commit i2b2/n2c2/MIMIC)
```

Verify your synthetic fixtures load and validate spans before you trust them:

```python
from openmed.eval.harness import load_fixtures

fixtures = load_fixtures("eval/gold/synthetic/test.json")
print(len(fixtures), "fixtures;", sum(len(f.gold_spans) for f in fixtures), "spans")
# load_fixtures normalizes spans against source text and rejects duplicate ids.
```

## Workflow

1. **Define the label schema.** Reuse OpenMed canonical labels (PERSON, DATE,
   ID_NUM, EMAIL, PHONE, DISEASE, DRUG, ...). Each label gets a one-line
   definition, in/out examples, and a boundary rule (include titles? trailing
   punctuation?).
2. **Write annotation guidelines.** The manual is the contract: span boundaries,
   nested/overlapping policy, ambiguous cases, and a decision log appended as
   real cases force calls. Vague guidelines → low agreement → unusable gold.
3. **Generate synthetic source text.** Compose realistic clinical narratives
   with *fabricated* identifiers (Faker-style names, impossible dates like
   2099-, all-zero MRNs). Never paste real notes into committed data.
4. **Configure the tool.** BRAT uses `annotation.conf` (entity types) producing
   `.ann` standoff; Label Studio uses a labeling-config XML producing JSON.
   Map either back to the fixture shape above.
5. **Double-annotate and measure agreement.** Have ≥2 annotators on an overlap
   set; compute span-level inter-annotator agreement (F1 or Cohen's κ). Adjudicate
   disagreements and fold the resolutions into the decision log.
6. **Split with discipline.** Partition **by document/patient**, not by sentence,
   so no patient appears in two splits. Freeze test; never tune on it.
7. **Validate and commit.** Run `load_fixtures`; confirm spans align and ids are
   unique. Commit only the synthetic splits.

## Hand-off to / from OpenMed

- **To** `benchmarking-clinical-ner`: dev/test fixtures feed `run_suite` and
  `error_report` for the NER scorecard.
- **To** `evaluating-with-leakage-gates` and `gating-deid-leakage`: the synthetic
  held-out set is exactly what the release gates and the CI gate run against.
- **To** `building-with-openmed`: synthetic notes can be generated by running
  surrogate replacement through `openmed.deidentify(method="replace")`.
- **Pairs with** `auditing-subgroup-fairness`: tag each gold span with a `group`
  in `metadata` so `fairness_report` can slice by demographic surrogate.

## Edge cases & gotchas

- **Committed = synthetic. No exceptions.** Real PHI in the repo is a breach even
  if the repo is private. Generate identifiers; don't transcribe them.
- **DUA data is eval-only.** Load i2b2/n2c2/MIMIC from `eval/external/` (gitignored)
  at runtime under the user's license; results may be reported, data never shared.
- **Split by patient, not by line.** Sentence-level splitting leaks a patient's
  style/identifiers across train and test and inflates scores.
- **Offsets must be character indices into *this* `text`.** Re-tokenization or
  whitespace edits silently shift offsets; re-validate with `load_fixtures`.
- **Label the fairness surrogate, not real demographics.** Put a synthetic
  `group` tag in span `metadata`; don't store real protected attributes.
- **Decision log is the gold's source of truth.** Without it, two re-annotations
  disagree and your "ceiling" F1 is noise.

## Standards & references

- BRAT rapid annotation tool & standoff format:
  https://brat.nlplab.org/ and https://brat.nlplab.org/standoff.html
- Label Studio labeling configuration:
  https://labelstud.io/tags/
- i2b2/n2c2 NLP shared tasks (annotation-guideline tradition; DUA-gated data):
  https://www.i2b2.org/NLP/ and https://n2c2.dbmi.hms.harvard.edu/
- MAE/MAI and inter-annotator agreement for span tasks (Cohen's κ, span F1):
  https://aclanthology.org/J96-2004/ (Carletta, on κ for NLP)
- OpenMed fixture loader: `openmed/eval/harness.py` (`load_fixtures`,
  `BenchmarkFixture.from_mapping`).