Text Cleaning And Extraction

---
name: text-cleaning-and-extraction
description: "Clean raw Chinese text, extract obvious patterns, and normalize
  noisy strings."
metadata:
  disco-role: operating
disable-model-invocation: true
license: Apache 2.0
---

# Text cleaning and extraction

Use this sub-skill for raw text cleanup, pattern extraction, replacement, sentence splitting, and file-line helpers.

## Include here
- `clean_text`, `clean_html`, `remove_html_tag`, `remove_exception_char`, `remove_redundant_char`, `convert_full2half`
- `extract_*`, `remove_*`, and `replace_*` helpers for email, URL, phone, IP, ID card, QQ, Chinese chars, parentheses, and vehicle plates
- `check_any_chinese_char`, `check_all_chinese_char`, `check_any_arabic_num`, `check_all_arabic_num`
- `split_sentence`
- `remove_stopwords` after tokenization
- `read_file_by_iter`, `read_file_by_line`, `write_file_by_line`, `TimeIt`, `zip_file`, `unzip_file`
- `jio.help()` discovery support

## Exclude or route elsewhere
- Semantic parsing of time, money, location, phone attribution, ID detail, plate interpretation, pinyin, radical, or idiom logic → `parsing-and-normalization`
- Data augmentation → `text-augmentation`
- Tag conversion, lexicon NER, dataset splits, or batching helpers → `annotation-and-dataset-tools`
- Dictionary loaders and higher-level language analysis → `dictionaries-and-language-analysis`

## What to read
- `references/api-reference.md` for exact function groups and return shapes.
- `references/troubleshooting.md` for boundary, HTML, file, and pattern-matching issues.
- `scripts/smoke_text_cleaning.py` for a fast validation run.

## Typical flow
1. Clean or normalize the text before any deeper parsing.
2. Extract or replace obvious entities.
3. Split or filter token lists only after tokenization.
4. Use the file helpers when the task is line-oriented data or temporary fixtures.

## Quick cues
- Ask for this sub-skill when the user says "remove URLs", "mask emails", "clean HTML", "full-width to half-width", "split sentences", or "search docs by keyword".
- Stay here if the task is only about surface text cleanup and not semantic interpretation.