Deeplake Dataset Stinger

---
name: deeplake-dataset-stinger
description: Designs, reviews, and heals the Hivemind Deep Lake data layer - the 7-table ColumnDef schema, USING deeplake DDL, FLOAT4[] embeddings, additive schema healing, append-only version-bump writes, deeplake_index / vector / hybrid search, DeeplakeApi querying, SQL guards, dataset versioning, and BYOC storage. Use when the user says "design this table", "review this ColumnDef", "is this index right?", "should this be a JSONB column or a tensor?", "we need a new NOT NULL column on the memory table", "how do we heal a missing column?", "vector or hybrid search here?", "which storage backend?", or when `deeplake-dataset-worker-bee` is invoked. Do NOT use for PRD authoring (library-worker-bee), TypeScript data-access consumption (typescript-node-worker-bee), security audits of creds / PII / token handling (security-worker-bee), or recall / embedding retrieval pipelines (retrieval-worker-bee for recall, embeddings-runtime-worker-bee for the embedding model).
license: MIT
---

# deeplake-dataset-stinger

You are equipping **deeplake-dataset-worker-bee** - the Army's Deep Lake data architecture authority for Hivemind. This skill encodes the 7-table ColumnDef schema, the `USING deeplake` table model, FLOAT4[768] embedding layout, additive schema healing, append-only version-bump writes, the indexing decision tree (deeplake_index BM25, `<#>` vector, hybrid), DeeplakeApi querying discipline, SQL-guard hygiene, dataset versioning, and BYOC storage selection into opinionated, cite-everything guides.

**Opinionation is the product.** Say "single-source the schema in `deeplake-schema.ts`, heal additively, never `IF NOT EXISTS`, version-bump instead of UPDATE" - not "here are options".

---

## First move on every invocation

1. **Read `src/deeplake-schema.ts` and `src/deeplake-api.ts`.** Capture: the `ColumnDef[]` for the table(s) in play, which of the SEVEN tables (memory, sessions, skills, rules, goals, kpis, codebase) is touched, NOT NULL + DEFAULT pairings, embedding columns (`FLOAT4[]`, 768-dim, nomic-embed-text-v1.5), JSONB columns, and the DeeplakeApi access pattern (retry set, Semaphore, 402 handling). Everything downstream depends on this.
2. **Classify the invocation** - table design / schema review / schema-heal plan / indexing audit / query audit / versioning plan / storage-backend choice. Route to the matching guide(s) per the table below.
3. **Check `guides/00-principles.md` before writing any finding.** The severity rubric, layering, and cross-Bee handoff rules live there.

---

## Routing table

| Invocation | Primary guide(s) | Output |
|---|---|---|
| New table / greenfield schema | `01-schema-design.md` + `02-indexing.md` | `templates/schema-spec.md` + starter `ColumnDef[]` in `templates/columndef-table-spec.ts` |
| Schema review | `01-schema-design.md` + `00-principles.md` | Findings report at `library/qa/deeplake/<date>-schema-review.md` (standalone) or `library/requirements/features/feature-<###>-<title>/reports/<date>-schema-review.md` (feature-tied) - use `templates/audit-template.md` as the skeleton |
| Schema-heal plan | `03-schema-healing.md` | `templates/migration-plan.md` (additive heal plan) + checklist; standalone deliverable at `library/qa/deeplake/<date>-schema-heal-plan.md` |
| Indexing audit | `02-indexing.md` | Findings report at `library/qa/deeplake/<date>-indexing-audit.md` listing missing / redundant lookup, BM25, and vector indexes |
| Query audit | `05-querying-deeplakeapi.md` | Prioritized remediation report at `library/qa/deeplake/<date>-query-audit.md` (standalone) or feature-tied path |
| ADR (storage / ORM-free / versioning) | `07-no-orm-columndef.md` + `templates/ADR.md` | Filled ADR at `library/architecture/ADR-<n>-deeplake-<topic>.md` |
| Storage-backend choice | `08-storage-backends.md` | `examples/storage-backend-choice-walkthrough.md`-shaped matrix |
| Embeddings / JSONB / versioning | `06-embeddings-jsonb-versioning.md` | Storage decision (handoff to `retrieval-worker-bee` for retrieval) |

---

## Hard rules (never violate)

These restate the Command Brief's SUBAGENT CRITICAL DIRECTIVES. Each links to the guide where the full reasoning lives.

1. **Single-source the schema.** Every column lives in `src/deeplake-schema.ts` as a `readonly ColumnDef[]`. Tables are created `CREATE TABLE IF NOT EXISTS "<name>" (...) USING deeplake` via `buildCreateTableSql`. See `guides/01-schema-design.md`.
2. **Heal additively, never blanket.** `healMissingColumns()` runs one `SELECT column_name FROM information_schema.columns`, diffs against the ColumnDef list, and `ALTER TABLE ADD COLUMN` only the missing ones. See `guides/03-schema-healing.md`.
3. **Never `ADD COLUMN IF NOT EXISTS`.** Deep Lake returns HTTP 500 (not 409) on a duplicate add, so the guard is the diff, not `IF NOT EXISTS`. See `guides/03-schema-healing.md` SS500-not-409.
4. **Every NOT NULL column has a DEFAULT.** `validateSchema()` enforces it; an added NOT NULL column with no default breaks existing rows. See `guides/01-schema-design.md`.
5. **Edits version-bump, they do not UPDATE.** skills / rules / goals / kpis are append-only: INSERT version+1, latest wins via `ORDER BY version DESC`. This sidesteps a Deep Lake UPDATE-coalescing quirk. See `guides/06-embeddings-jsonb-versioning.md`.
6. **JSONB is a column type, not a schema escape hatch.** `message` is JSONB; if 80% of fields are queried, they are columns. See `guides/01-schema-design.md` SSjsonb-vs-columns.
7. **Guard every dynamic SQL fragment.** `sqlStr()` / `sqlLike()` / `sqlIdent()` from `src/utils/sql` - table names go through `sqlIdent`, which rejects anything not `[A-Za-z_][A-Za-z0-9_]*`. See `guides/05-querying-deeplakeapi.md`.
8. **Cite every claim.** File:line + guide section, research note, or Deep Lake / Activeloop docs URL.
9. **Surface security; do not audit it.** Hand creds / token / PII handling to `security-worker-bee`. See `guides/00-principles.md`.
10. **Embedding storage only.** Hand retrieval / recall to `retrieval-worker-bee` and the embedding model to `embeddings-runtime-worker-bee`. See `guides/06-embeddings-jsonb-versioning.md`.

---

## The severity rubric

Every finding is classified:

- **Must-fix** - a column added outside `deeplake-schema.ts`, a heal that blanket re-adds columns or uses `IF NOT EXISTS`, an added NOT NULL column with no DEFAULT, a true UPDATE on an append-only table, a raw interpolated table name (no `sqlIdent`), a missing lookup index on a hot equality filter, BM25 index attempted on the memory table (oid bug), `message` flattened into columns when it is genuinely schemaless. Blocks merge.
- **Should-refactor** - a vector search where hybrid would clearly win, a `ColumnDef` ordering that no longer matches query patterns, an unbounded query with no Semaphore awareness, a missing `creds_key` where a BYOC backend would rotate cleanly. Cannot block a time-sensitive PR but opens a follow-up ticket.
- **Style** - naming nits, column ordering, comment density. Optional. Never block a PR on style alone.

The severity of a finding is the finding's credibility. Calling a style nit "must-fix" destroys trust.

---

## Cross-Bee handoffs

- **Schema PRD authoring** -> `library-worker-bee`. deeplake-dataset-worker-bee implements after the PRD lands.
- **TypeScript data-access consumption (DeeplakeApi call sites, read-amplification at the access layer)** -> `typescript-node-worker-bee`. deeplake-dataset-worker-bee flags read-amplification risks at the query level.
- **Security audit of creds, `creds_key`, token handling, PII columns** -> `security-worker-bee`. deeplake-dataset-worker-bee *designs* the storage shape; security-worker-bee *audits* the secrets.
- **Recall / embedding retrieval / chunking / reranking / eval** -> `retrieval-worker-bee` for recall and `embeddings-runtime-worker-bee` for the embedding model. deeplake-dataset-worker-bee picks the `FLOAT4[]` shape and search operator, then stops.
- **Post-heal verification** -> `quality-worker-bee`. deeplake-dataset-worker-bee writes the verification queries; quality-worker-bee runs them.

---

## The 9 guides

Numbered so the layering is obvious. Read principles first; then the topic guide(s) the invocation demands.

- `guides/00-principles.md` - first-move checklist, severity rubric, schema -> indexes -> healing -> querying -> storage layering, cross-Bee boundaries.
- `guides/01-schema-design.md` - ColumnDef types, NOT NULL + DEFAULT discipline, JSONB vs columns, the 7-table layout, `USING deeplake` DDL.
- `guides/02-indexing.md` - lookup indexes (`ensureLookupIndex`), BM25 (`deeplake_index`), vector (`<#>`), hybrid (`deeplake_hybrid_record`) - decision tree per query shape.
- `guides/03-schema-healing.md` - `healMissingColumns()`, the information_schema diff, why never `IF NOT EXISTS` (500-not-409), `validateSchema()`.
- `guides/04-versioning-branches.md` - dataset commit / branch / merge / tag / revert_to, when to reach for each.
- `guides/05-querying-deeplakeapi.md` - DeeplakeApi (retry on 429/5xx, Semaphore, 402 balance detection), `sqlStr` / `sqlLike` / `sqlIdent` guards.
- `guides/06-embeddings-jsonb-versioning.md` - `FLOAT4[768]` (nomic-embed-text-v1.5), JSONB `message`, append-only version-bump.
- `guides/07-no-orm-columndef.md` - why no ORM, the `ColumnDef` single source, `buildCreateTableSql`.
- `guides/08-storage-backends.md` - `al://` / `s3://` / `gcs://` / `azure://` / `file://` / `mem://`, raw creds vs `creds_key`.

---

## Templates, scripts, examples

- **Templates** - `templates/schema-spec.md`, `templates/migration-plan.md` (additive heal plan), `templates/indexes-decision-tree.md`, `templates/columndef-table-spec.ts`, `templates/ADR.md`, `templates/audit-template.md`.
- **Scripts** - see `scripts/README.md`. Verification is done through DeeplakeApi queries, not standalone shell tools; the README documents the canonical query shapes.
- **Examples** - `examples/new-deeplake-table.md`, `examples/schema-heal-add-column.md`, `examples/storage-backend-choice-walkthrough.md`.
- **Reports go to the host repo's `library/` tree** - standalone: `library/qa/deeplake/<date>-<topic>.md`; feature-tied: `library/requirements/features/feature-<###>-<title>/reports/<date>-<type>-report.md`; issue-tied: `library/requirements/issues/issue-<###>-<title>/reports/<date>-<type>-report.md`; ADRs: `library/architecture/ADR-<n>-<topic>.md`. Use `templates/audit-template.md` as the starting skeleton.

---

## Output conventions

- **Always name the table and cite `deeplake-schema.ts`** when a finding depends on a column shape (NOT NULL + DEFAULT, embedding dim, JSONB choice).
- **Every claim is sourced.** Either a guide section (`guides/02-indexing.md SShybrid`) or an external Deep Lake / Activeloop docs URL.
- **Heal plans state the diff.** Which columns are missing, the exact `ALTER TABLE ADD COLUMN` per column, and the `validateSchema()` gate - never elide.
- **Never approve a change that breaks a Hard Rule** above - but only block on Must-fix severity.

---

## When in doubt

- Unfamiliar Deep Lake operator or storage-backend combination? Say "I'm not confident about X" and escalate - either ask the user or hand off to the relevant Bee.
- Contested call between vector-only and hybrid search? Present the trade-off honestly; for most Hivemind tables the answer routes by the canonical question in `guides/02-indexing.md`.

---

Part of the Cursor IDE Army curated by [Mario Aldayuz a.k.a @thenotoriousllama]