Vdjdb Format

---
name: vdjdb-format
description: Standardise a raw or partially-formatted VDJdb TSV chunk — normalising species names, IMGT V/D/J gene IDs, IMGT-HLA MHC alleles, and method vocabulary — and produce a properly-named chunk file ready for proofreading.
---

# /format — VDJdb Chunk Formatting Skill

## Purpose

Take the output of `/extract` (or any TSV resembling a chunk file) and standardise all controlled-vocabulary fields to the VDJdb / IMGT specification. Cross-check naming conventions against existing `chunks/` files to ensure internal consistency. This is the second stage: extract → **format** → proofread.

## Invocation

```
/format [path-to-tsv]
```

The TSV must have a VDJdb-compatible header (see canonical column order in `/extract`). If the file has structural problems (missing columns, wrong separator), halt and report — this is a job for `/proofread` Step 1, not format.

---

## Standardisation Rules

### 1. Species Names

Normalise to the exact VDJdb-accepted values (case-sensitive):

| Normalise FROM | Normalise TO |
|---|---|
| `Homo sapiens`, `human`, `H. sapiens`, `hs`, `Human` | `HomoSapiens` |
| `Mus musculus`, `mouse`, `M. musculus`, `mm`, `Mouse` | `MusMusculus` |
| `Rattus norvegicus`, `rat`, `Rattus` | `RattusNorvegicus` |
| `Macaca mulatta`, `rhesus`, `macaque`, `NHP` | `MacacaMulatta` |

If a species is not in the above list:
1. Log it as a candidate for extending `speciesList` in `py_src/ChunkQC.py`
2. Ask the user whether to include or exclude those rows

---

### 2. IMGT V/D/J Gene IDs

**Primary authority:** `proofreading/imgt_alleles.tsv.gz` (column `imgt_gene_id`)
**Conversion table:** `patches/nomenclature.conversions`
**Secondary fallback:** `patches/IGM_nomenclature_table.tsv`

**Rules (apply in order):**

1. **Strip whitespace**: remove all spaces within the gene name (`TRBV 7` → `TRBV7`, `TRAV 12-2` → `TRAV12-2`)

2. **Detect and convert Adaptive Biotech ImmunoSEQ names** (see `proofreading/imgt.md` §9.2 for full details):
   - **Full Adaptive prefix** (`TCRB`, `TCRA`, `TCRG`, `TCRD`): replace with `TRB`, `TRA`, `TRG`, `TRD`
     - `TCRBV06-05*01` → strip `TCR` prefix → `TRBV06-05*01`
   - **Zero-padded subgroup**: strip leading zeros from subgroup number
     - `TRBV06-5` → `TRBV6-5`
   - **Zero-padded cluster**: strip leading zeros from cluster number
     - `TRBV7-06` → `TRBV7-6`, `TRBV4-01` → `TRBV4-1`
   - **Verify result in `imgt_alleles.tsv.gz`**: if `gene-cluster` is not found, try the bare gene name (Adaptive always appends `-01` to single-cluster genes that IMGT names without a cluster)
     - `TRBV19-01` → `TRBV19-1` not found → `TRBV19` found ✓
     - `TRBV11-02` → `TRBV11-2` found ✓
   - When source is Adaptive, note in format log: `ADAPTIVE_NAME → IMGT_NAME (Adaptive ImmunoSEQ normalisation)`

3. **Look up in `imgt_alleles.tsv.gz`** (strip allele suffix `*NN` before lookup):
   - If found → keep (or correct capitalisation to match)
   - If not found → check `patches/nomenclature.conversions` for a mapping
   - If found in conversions → apply the conversion and log `old_name → new_name`
   - If not found in either → flag as unresolvable; ask user

3. **Validate allele** (if present, e.g., `TRBV12-3*02`):
   - Look up the full allele name in `imgt_allele_id` column: `gzip -dc proofreading/imgt_alleles.tsv.gz | awk -F'\t' '$3=="TRBV12-3*02"'`
   - If not found as a complete allele: flag as invalid; check whether the gene itself exists (gene-level lookup)

4. **Check functionality**: if `functionality` is `P` (pseudogene) in `imgt_alleles.tsv.gz`: flag as biologically suspicious

5. **Consistency check against existing chunks**:
   ```bash
   grep -h "" chunks/*.txt | cut -f3 | sort -u | grep "^TRAV"  # check v.alpha values
   ```
   If the same gene appears with different notation in existing chunks (e.g., `TRAV13-1` vs `TRAV13`), standardise to the IMGT-canonical form.

6. **Multiple gene possibilities** (comma-separated ambiguous assignments): check each against `imgt_alleles.tsv.gz`, keep all valid candidates comma-separated without spaces (e.g., `TRBV7-2,TRBV7-3`)

---

### 3. MHC Alleles

**Primary authority:** `proofreading/mhc_alleles.tsv.gz` (column `allele_name`)
**Reference:** `proofreading/mhc.md`

#### Human MHC (HLA)

Target format: `HLA-<GENE>*<FIELD1>:<FIELD2>` (e.g., `HLA-A*02:01`)

| Problem | Fix |
|---|---|
| `A02`, `A0201` (old serological) | → `HLA-A*02:01` if unambiguous; flag if ambiguous |
| `A*0201` (old format, no colon) | → `HLA-A*02:01` (add prefix + insert colon) |
| `HLA-A*02` (low resolution, 1-field) | Keep as-is; note in log that higher resolution preferred |
| `HLA-A*02:01:01` or `*02:01:01:01` (high-res) | Keep full string as-is |
| `HLA-A 02:01` (space) | → `HLA-A*02:01` |
| `HLA-A*02:01N`, `*02:01L` (expression suffix) | Keep suffix; note in log |
| Confirmed status from `mhc_alleles.tsv.gz` | `gzip -dc proofreading/mhc_alleles.tsv.gz \| awk -F'\t' '$2=="HLA-A*02:01"{print $3}'` |

**MHC-I second chain**: always normalise to literal `B2M` — never `beta-2-microglobulin`, `β2m`, `b2m`, `B2M*01`, etc.

**mhc.class cross-check:**
| If `mhc.a` starts with... | `mhc.class` must be | `mhc.b` must be |
|---|---|---|
| `HLA-A`, `HLA-B`, `HLA-C`, `HLA-E`, `HLA-F`, `HLA-G` | `MHCI` | `B2M` |
| `HLA-DR`, `HLA-DQ`, `HLA-DP`, `HLA-DO` | `MHCII` | HLA β-chain allele |

#### Mouse MHC (H-2)

| Normalise FROM | Normalise TO |
|---|---|
| `H2-Db`, `H2Db` | `H-2Db` |
| `IAb`, `I-Ab`, `IA-b` | `I-Ab` |
| `H-2D^b` | `H-2Db` |

For mouse class I: `mhc.b = B2M`
For mouse class II: `mhc.b` = the β-chain name (e.g., `I-Ab`)

---

### 4. Method Vocabulary

Normalise `method.identification` and `method.verification` to VDJdb-recognised terms.

**The governing rule: use what the source says.** Do not upgrade or downgrade based on prevalence in VDJdb.

| Author writes | Normalise to | Reasoning |
|---|---|---|
| `tetramer`, `pMHC tetramer`, `tetramer sort` | `tetramer-sort` | Source specifies tetramers |
| `dextramer`, `dextramer sort` | `dextramer-sort` | Source specifies dextramers |
| `pentamer`, `pentamer sort` | `pentamer-sort` | Source specifies pentamers |
| `multimer`, `pMHC multimer`, `multimer sort` | `multimer-sort` | Source gives no more specific reagent type |
| Reagent type not stated (only "sort" or "FACS") | `multimer-sort` | Cannot assume tetramer; use generic |
| `ELISpot` | Do NOT map automatically | Log and ask user |
| `51Cr release assay` | Do NOT map automatically | Log and ask user |

> **Example:** A readme that says only "tetramer-sort" → `tetramer-sort`. A readme that says only "multimer-sort" with no other information → `multimer-sort`, even if tetramers are the most common reagent in VDJdb. Never infer the reagent type from context or database prevalence.

**Rule:** If an identification or verification method has no close equivalent in the current VDJdb vocabulary, do NOT force it. Instead:
1. Leave a descriptive string in the field (for reference)
2. Document it under "Vocabulary gaps" in the format log
3. Suggest adding it as a new term via a note to the database maintainers

---

### 5. Reference IDs

Enforce correct format:

| Problem | Fix |
|---|---|
| `https://doi.org/10.1016/...` | → `doi:10.1016/...` (remove URL prefix) |
| `http://dx.doi.org/10.1016/...` | → `doi:10.1016/...` |
| `doi: 10.1016/...` (space after colon) | → `doi:10.1016/...` |
| `pubmed:12345678` or `PubMed:12345678` | → `PMID:12345678` |
| Bare number `12345678` | Ask if it is a PMID; if confirmed → `PMID:12345678` |

---

### 6. Antigen Fields

Cross-reference `patches/antigen_epitope_species_gene.dict`:
- If `antigen.epitope` exists in the dict → use the dict's `antigen.gene` and `antigen.species` (this ensures consistency with the full database)
- If the epitope is new → keep author-provided gene/species values, note in log

---

### 7. Chunk ID

After all formatting changes, renumber `chunk.id` sequentially from 1 (integer, no leading zeros).

---

## Output Filename

Prefer `PMID_<pubmed_id>.txt` (e.g., `PMID_28975614.txt`).

If no PMID is available:
1. Ask the user for the preferred name
2. Alternatives: `doi_<mangled_doi>.txt`, submitter-date format
3. Check that the chosen name does not duplicate an existing file in `chunks/`

Suggested placement: `chunks_unformatted/` if uncertain about QC status; `chunks/` only after `/proofread` passes.

---

## Format Log

Write `<output_basename>_format_log.txt` containing:

1. **Changes made**: for each change — field name, old value, new value, source of normalisation (imgt_alleles.tsv.gz / mhc_alleles.tsv.gz / nomenclature.conversions / manual)
2. **Unresolvable fields**: fields that could not be normalised and why
3. **Vocabulary gaps**: novel method/verification terms encountered
4. **Allele resolution notes**: alleles that exist in `mhc_alleles.tsv.gz` at low resolution only
5. **Consistency discrepancies**: naming differences found vs existing `chunks/` files
6. **Pseudogene warnings**: gene names whose `functionality = P` in `imgt_alleles.tsv.gz`
7. **Unconfirmed HLA alleles**: alleles present in `mhc_alleles.tsv.gz` with `confirmed = Unconfirmed`

---

## Reference Files

| File | Role |
|---|---|
| `proofreading/imgt_alleles.tsv.gz` | **Primary** IMGT V/D/J gene authority |
| `proofreading/imgt.md` | IMGT nomenclature rules |
| `proofreading/mhc_alleles.tsv.gz` | **Primary** HLA allele authority |
| `proofreading/mhc.md` | MHC/HLA naming rules, class I vs II, non-human |
| `patches/nomenclature.conversions` | Old → current IMGT gene name mappings |
| `patches/IGM_nomenclature_table.tsv` | Secondary IMGT fallback (existing repo file) |
| `patches/antigen_epitope_species_gene.dict` | Known epitope → gene/species mappings |
| `py_src/ScoreFactory.py` | Method vocabulary and scoring logic |
| `py_src/ChunkQC.py` | ALL_COLS definition (canonical column list) |
| `chunks/*.txt` | Reference for consistency checks |
| `README.md` | Full VDJdb specification |

---

## Next Step

After formatting, run `/proofread [path-to-formatted-tsv]` to validate against `py_src/ChunkQC.py` and all other QC checks.