Pipeline Components

---
name: "pipeline-components"
description: "Routes scispaCy custom tokenizer, sentence segmentation,
  abbreviation detection, hyponym detection, and whitespace-tokenization
  workflows."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Pipeline Components

Use this sub-skill when the task is about scispaCy's biomedical text-processing components rather than knowledge-base linking or project data workflows.

## Typical triggers

- Add or debug `abbreviation_detector`.
- Add or debug `hyponym_detector`.
- Replace spaCy's tokenizer with `combined_rule_tokenizer`.
- Use `pysbd_sentencizer` for sentence boundaries in biomedical text.
- Use `WhitespaceTokenizer` for pretokenized BIO-style inputs.
- Build or debug a biomedical pipeline around `en_core_sci_sm` or another scispaCy model.

## What belongs here

- Custom tokenization rules and punctuation handling.
- Sentence segmentation with `pysbd`.
- Abbreviation expansion and serialization.
- Hearst-pattern hyponym extraction.
- Whitespace tokenization for already tokenized text.
- The combined-rule model helper used in small pipeline smoke checks.

## What does not belong here

- Entity-linker / KB / ANN-index work. Use `entity-linking`.
- MedMentions readers, evaluation scripts, and UMLS export. Use `project-workflows`.
- Package installation and model catalog details. Read the root installation reference.

## First things to read or run

- `references/api-reference.md` for verified signatures, factory names, and extension attributes.
- `references/workflows.md` for concrete pipeline assembly patterns.
- `references/troubleshooting.md` for registration, serialization, and segmentation failures.
- `../../scripts/smoke_scispacy.py` when you need a quick install or component smoke.

## Fast workflow summary

1. Import the factory-registration modules once in the process:
   - `import scispacy.abbreviation`
   - `import scispacy.hyponym_detector`
2. Load a spaCy model such as `en_core_sci_sm` for biomedical text or `en_core_web_sm` for general-English tokenization checks.
3. Replace the tokenizer or add the sentence-segmentation pipe before downstream components if the text needs custom rules.
4. Add `abbreviation_detector` before `scispacy_linker` when the linker should resolve abbreviations.
5. Use `make_serializable=True` when the document must survive `to_bytes()` or multiprocessing.

## Example route decisions

- If the request says "make the tokenizer split biomedical hyphens correctly", stay here.
- If the request says "detect abbreviations and serialize the doc", stay here.
- If the request says "link entities to UMLS", switch to `entity-linking`.
- If the request says "evaluate NER on MedMentions", switch to `project-workflows`.

## Verification mindset

For this sub-skill, the strongest validation is a small pipeline smoke:

- `abbreviation_detector` finds the long form for a simple abbreviation pair.
- `hyponym_detector` finds a Hearst-pattern match such as `such_as`.
- `WhitespaceTokenizer` preserves pretokenized input.
- A combined pipeline can serialize when abbreviation detection is configured for serialization.

If one of those checks fails, read the troubleshooting reference before changing the broader environment.