Dictionaries And Language Analysis

---
name: dictionaries-and-language-analysis
description: "Load packaged dictionaries and use JioNLP's keyphrase, summary,
  sentiment, new-word, BPE, and evaluation helpers."
metadata:
  disco-role: operating
disable-model-invocation: true
license: Apache 2.0
---

# Dictionaries and language analysis

Use this sub-skill when the task is about packaged resources, corpus statistics, or higher-level language-analysis helpers built on top of JioNLP's dictionaries.

## Include here
- Packaged loaders for stopwords, locations, pinyin, radicals, distributions, sentiment words, quantifiers, and the LLM test set
- `jio.keyphrase.extract_keyphrase`, `jio.summary.extract_summary`, `jio.sentiment.LexiconSentiment`
- `jio.new_word.new_word_discovery`
- `jio.text_classification.analyse_freq_words`
- `jio.text_classification.analyse_dataset` when the task is about label distribution and class splits
- `jio.bpe.byte_level_bpe`
- `jio.mellm.MELLM`

## Exclude or route elsewhere
- Raw cleanup or extraction → `text-cleaning-and-extraction`
- Semantic parsing of time, money, location, or IDs → `parsing-and-normalization`
- Augmentation → `text-augmentation`
- CWS/POS/NER conversion and batching → `annotation-and-dataset-tools`

## What to read
- `references/api-reference.md` for the callable helpers.
- `references/loader-catalog.md` for packaged resource shapes.
- `references/llm-dataset.md` for the LLM test-set loader and MELLM notes.
- `references/troubleshooting.md` for packaged-data and network issues.
- `scripts/smoke_language_tools.py` for a safe smoke run.

## Typical flow
1. Load the packaged resource or sample corpus you need.
2. Decide whether the helper is frequency-based, topic-based, or evaluation-based.
3. For LLM evaluation, load the test dataset first and then wire the API callables.
4. Use the bundled smoke script to confirm the package resources are readable.

## Quick cues
- Ask for this sub-skill when the user says "load the dictionary", "extract keywords", "summarize this text", "score sentiment", "find new words", "use the LLM test set", or "run MELLM".
- Stay here if the task depends on packaged resources, topic statistics, or corpus-level language analysis.