Collocation Morphology Preprocessor

---
name: collocation_morphology_preprocessor
description: Preprocess English text with WordNet-style collocation matching, tokenization, and inflectional normalization.
---

# Collocation and Morphology Preprocessor

Use this skill when a recovery or application needs the preprocessing behavior described in the WordNet report: one sentence per line, WordNet collocation search, tokenization, part-of-speech hints, and morphology-aware lookup forms.

## Inputs
- Raw English text.
- A list of known collocations, using spaces or underscores.

## Outputs
- Sentence records with original text, token records, normalized lemmas, POS hints, and collocation spans.

## Workflow
1. Split bounded text into sentence records.
2. Lowercase and remove punctuation for matching while preserving original sentence text.
3. Greedily match known collocations before single-token normalization.
4. Normalize regular plurals and common `-ing` or `-ed` inflections.
5. Emit unknown tokens rather than dropping them, because the ConText-style tagger must record unresolved reasons.

## Validation
Run `python tests/test_preprocessor.py` or the Distiller skill-tree validator with tests enabled.

## Limitations
The POS hints are deterministic and lightweight. They are sufficient for reduced recovery but not a replacement for a full part-of-speech tagger.