Integrations And Storage

---
name: integrations-and-storage
description: "Export Chonkie chunks to files/datasets and use vector datastore
  handshakes safely."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# Integrations and Storage

Use this sub-skill when the task is about persisting, exporting, or handing off
Chonkie chunks/documents after chunk creation:

- write chunks as JSON/JSONL with `JSONPorter`;
- create or save Hugging Face `Dataset` objects with `DatasetsPorter`;
- route `Pipeline.export_with(...)` and `Pipeline.store_in(...)` steps;
- prepare Chonkie vector/datastore handshakes without accidental live writes;
- diagnose optional Chroma, Qdrant, LanceDB, Milvus, MongoDB, Pgvector,
  Pinecone, Turbopuffer, Weaviate, or Elasticsearch dependency issues.

## First routing decisions

1. If the user has not created chunks yet, load `../chunking-and-types/` for
   chunk/data-shape guidance and `../pipelines-and-processing/` for CHOMP
   pipeline composition before choosing storage.
2. If a vector datastore handshake needs `embedding_model`, load
   `../embeddings-and-generative/` before selecting the model, dimensions, or
   provider credentials.
3. If the user asks for CLI syntax, handshaker flags, serving, or deployment
   wiring, load `../interfaces-and-deployment/` and then return here for
   storage-specific constructor arguments and safety gates.
4. Prefer local, no-network exports (`JSONPorter` or `DatasetsPorter` with a
   caller-chosen path) unless the user explicitly asks for a datastore.
5. For vector/datastore integrations, run `scripts/handshake_dependency_probe.py`
   first when dependency availability is uncertain. Do not instantiate a live
   datastore client or write chunks until the target service, credentials, and
   write scope are explicit.

## Reference map

- `references/porter-workflows.md` — JSON/JSONL and Hugging Face Dataset
  workflows, pipeline export examples, and output-shape notes.
- `references/handshakes-reference.md` — BaseHandshake contract, dependency
  extras, constructor argument tables, random naming behavior, embedding model
  gates, search outputs, and safe mocked/in-memory alternatives.
- `references/troubleshooting.md` — import failures, credential/service safety,
  embedding and vector-dimension problems, metadata quirks, duplicate writes,
  and pipeline parameter mistakes.
- `scripts/handshake_dependency_probe.py` — no-network probe for optional client
  packages and Chonkie handshake/porter class imports with installation hints.

## Safety defaults

- Treat every handshake write as a datastore mutation. Avoid live writes by
  default; use mocked clients, in-memory clients, or local temporary paths for
  usability tests.
- Do not rely on default constructors for Milvus, MongoDB, Pgvector, Weaviate,
  or Elasticsearch unless the user confirms a local service is intentionally
  running and writable.
- Do not use Pinecone or Turbopuffer with real credentials unless the user
  confirms the target index/namespace and accepts remote writes.
- Prefer explicit collection/index/table/namespace names for repeatable work.
  Use `"random"` only for isolated experiments where discoverability is not
  required.
- Never paste secrets into examples. Read credentials from the user's approved
  secret manager or environment only when a live-service run is explicitly
  authorized.