Claude Skills by databricks-solutions

Canonical pattern for building an agentic tool-calling loop using a Databricks-hosted Foundation Model. Use when creating agent_loop.py or any backend that calls a Foundation Model with function-calling to orchestrate tools.

Entry-point navigator for Databricks AppKit + Lakebase development. Routes tasks to the correct skill based on keyword detection: scaffolding, plugin integration, feature building, or deployment. This is a routing skill -- it does not generate code. It directs the agent to the correct specialized skill. Use this skill as the starting point for any AppKit-related task. Triggers on "AppKit", "Lakebase app", "Databricks app", "build app", "deploy app", "scaffold app", "add plugin", "AppKit proje...

Scaffold new Databricks AppKit applications using the Databricks CLI and Agent Skills. Creates blank or plugin-enabled AppKit projects (Lakebase, Analytics, Genie, Files). Use when asked to create a Databricks app, scaffold an AppKit project, bootstrap a new app, or set up a full-stack TypeScript Databricks application. Triggers on "create app", "new app", "scaffold", "AppKit", "databricks app", "blank app", "bootstrap app", "init app". To add a plugin to an existing app, use the 04-appkit-pl...

Build full-stack UI and backend features on a Databricks AppKit project from a PRD or feature spec. Covers SQL query design, type generation, React frontend with AppKit UI components, backend plugin wiring, and distinctive visual design. Use when asked to implement a UI, build features from a PRD, create pages or dashboards, add components, develop the frontend/backend of an existing AppKit app, build with mock data, or use static data for visual prototyping. Triggers on "build UI", "implemen...

Deploy a Databricks AppKit application to Databricks Apps. Covers config validation, build verification, deployment, UI verification, error diagnosis with fix loop, and workspace app limit handling. Use when asked to deploy an AppKit app, push to production, ship the app, or troubleshoot a failed deploy. Triggers on "deploy app", "push to databricks", "ship app", "deploy appkit", "databricks apps deploy", "fix deploy error", "app won't start".

Add plugins to an existing Databricks AppKit project. Covers Lakebase (PostgreSQL), Analytics (SQL queries + dashboards), Genie (natural language AI/BI), Files (UC Volumes), and Serving (Model Serving / Agent endpoints). Guides through plugin registration, environment variables, app.yaml configuration, and frontend integration. Use when asked to add a plugin to an existing app, integrate Lakebase, add analytics, connect a Genie space, enable file uploads, add a serving endpoint, or extend an ...

Wire a Lakebase PostgreSQL backend into an existing AppKit project. Covers database schema design from a PRD, idempotent DDL, Express API routes with mock fallback, React data hooks, and local testing. PRD-independent patterns that apply to any AppKit + Lakebase app. Use after registering the Lakebase plugin via 04-appkit-plugin-add. Triggers on "wire lakebase", "lakebase backend", "CRUD API", "lakebase tables", "DDL", "database schema design", "useLakebaseData", "mock fallback", "ConnectionS...

Wire a Databricks Model Serving or Agent endpoint into an existing AppKit project using the Serving plugin. Covers endpoint registration, app.yaml resource binding, streaming and invoke React hooks, conversation state management, agent response mapping, and server-side proxy patterns. PRD-independent patterns that apply to any AppKit + Serving app. Use after registering the Serving plugin via 04-appkit-plugin-add. Triggers on "wire agent", "agent endpoint", "serving plugin", "agent UI", "chat...

Wire an AppKit frontend (Databricks App) to a separate Agent App backend (Databricks App hosting mlflow.genai.agent_server or ResponsesAgent) with OBO token forwarding via x-forwarded-access-token, app-to-app service principal auth, dual-format SSE streaming, and the /api/chat contract. Use this skill when the agent is deployed as its own Databricks App — not as a Model Serving endpoint. Triggers on "agent app backend", "two apps", "appkit plus agent app", "proxy agent app", "agent-backend re...

Add persistent chat history to an AppKit application using the Lakebase plugin and the Serving plugin (both AppKit-native, no Vercel AI SDK). Covers database schema, session auth middleware, a server-side proxy route that streams from `AppKit.serving("agent").asUser(req).stream()` while persisting assistant messages and capturing MLflow `trace_id`, the history API routes, a history sidebar, and ephemeral-mode fallback. Use when asked to persist conversations, add chat history, save messages, ...

Add user feedback (thumbs up/down) to an AppKit chat application, linked to MLflow assessments via the Databricks Assessments REST API. Covers the Vote table, feedback API routes (with AppKit-native auth via `getExecutionContext().client.config.authenticate()`), MLflow trace integration, and feedback UI components. Use when asked to add feedback, thumbs up/down, ratings, or link user judgments to MLflow traces. Triggers on "feedback", "thumbs up", "thumbs down", "rate response", "MLflow asses...

Guides the creation of new Agent Skills following the AgentSkills.io specification. Covers required frontmatter (name, description), optional metadata, directory structure (SKILL.md, references/, scripts/, assets/), progressive disclosure principles, and naming conventions. Use when creating a new skill from scratch, adding project-specific agent knowledge, defining reusable patterns for AI coding assistants, or structuring domain expertise as a skill. Triggers on "create skill", "new skill",...

Enforces documentation organization standards and provides comprehensive framework documentation authoring for Databricks projects. Operates in two modes: (1) Organizational Enforcement — validates file placement (root directory rules, docs/ hierarchy), naming conventions (kebab-case, date-prefixed), and proactively suggests cleanup for misorganized files. (2) Framework Documentation — orchestrates creation of complete technical documentation sets using fill-in-the-blank templates (index, int...

Enables agent self-reflection and learning from mistakes through systematic skill updates. Prioritizes updating existing skills over creating new ones - always searches existing skills first, creates new skills only when justified. Includes upstream source sync workflow for tracking and updating skills from databricks/databricks-agent-skills. Use after encountering errors, completing complex tasks, or when asked to reflect, learn, or document a mistake. Triggers on "learn from this", "don't r...

Systematic skill freshness auditing with verification anchors, volatility classification, staleness detection, and upstream lineage tracking across every Agent Skill in the repository (data_product_accelerator/, genai-agents/, apps_lakebase/). Fetches official documentation URLs embedded in skills, compares patterns against live docs, tracks lineage to databricks/databricks-agent-skills (the single upstream registry), and reports drift. Use when auditing skill currency, verifying skills again...

End-to-end Bronze layer creation for testing and demos. Creates table DDLs, generates fake data with Faker, copies from existing sources, and configures Asset Bundle jobs. Covers Unity Catalog compliance, Change Data Feed, automatic liquid clustering, and governance metadata. Use when setting up Bronze layer tables, creating test/demo data, rapid prototyping Medallion Architecture, or bootstrapping a new Databricks project. For Faker-specific patterns (corruption rates, function signatures, p...

Generate synthetic data with Faker for Bronze layer testing with configurable data corruption. Use when creating test data for data quality validation, testing DLT expectations, or simulating production-like datasets. Supports realistic data generation with intentional corruption patterns mapped to specific DQ expectations.

End-to-end autonomous deployment and operations skill. Deploys Databricks Asset Bundles, runs jobs and pipelines, polls for completion, diagnoses failures, applies fixes, redeploys, and verifies — all without human intervention. Also serves as SDK/CLI/REST API reference. Use when deploying bundles, running jobs, monitoring pipelines, troubleshooting ANY failure, or operating in a self-healing deploy-fix-redeploy cycle. Triggers on "deploy", "bundle deploy", "run job", "run pipeline", "make it...

Patterns for sharing code between Databricks notebooks using pure Python files and standard imports. Covers Asset Bundle path setup (rsplit canonical pattern), notebook-to-module conversion, import patterns vs %run magic commands, job submission context vs notebook context, MLflow model packaging path requirements, and troubleshooting ModuleNotFoundError. Use when creating shared modules, deploying jobs that import local code, or packaging MLflow models with local dependencies.

Provides standard TBLPROPERTIES and metadata patterns for Unity Catalog Delta tables across Bronze, Silver, and Gold medallion layers. Ensures governance compliance, performance optimization, and proper metadata tagging for all table creation operations. Covers required TBLPROPERTIES by layer (Bronze, Silver DLT, Gold), mandatory CLUSTER BY AUTO configuration, Change Data Feed (CDF) enablement, auto-optimize settings, table and column comment patterns (LLM-friendly for Bronze/Silver, dual-pur...

Enforces enterprise naming conventions (snake_case, table prefixes, approved abbreviations), dual-purpose COMMENT formats for tables/columns/TVFs/metric views/dashboards/Genie Spaces, and config-aware tagging standards. Scans context/ for customer tagging standards in any format (YAML, CSV, Markdown, JSON, TXT); derives meaningful smart defaults when none supplied. Uses Databricks Data Classification class.* system governed tags for PII (always inferred from column names + customer declaratio...

Provides schema management patterns for Databricks Asset Bundles with Unity Catalog. Enables programmatic schema creation and configuration for medallion architecture layers. Covers CREATE SCHEMA IF NOT EXISTS patterns, DLT pipeline schema configuration, enabling Predictive Optimization at schema level using ALTER SCHEMA ENABLE PREDICTIVE OPTIMIZATION (not TBLPROPERTIES), schema variable usage, and common pitfalls to avoid. Use when creating Unity Catalog schemas programmatically in setup scr...

Unity Catalog Primary Key and Foreign Key constraint patterns for proper relational modeling in Databricks. Use when implementing star schema dimensional models with PK/FK relationships in Gold layer tables. Covers surrogate keys as PRIMARY KEYS (not business keys), facts referencing surrogate PKs via FOREIGN KEY constraints, NOT NULL requirements for PK columns, proper dimensional modeling patterns (SCD Type 1/2, date dimensions), production deployment error prevention (never define FK inlin...

Create dual-format ad-hoc exploration notebooks for Databricks workspace (.py) and local Jupyter (.ipynb) with Databricks Connect. Use when building data exploration tools, debugging data quality issues, or creating interactive analysis notebooks. Supports widget fallback patterns, helper functions for table discovery, and proper Spark session initialization for both environments.

Scaffold a minimal MCP tool-calling agent with Genie Spaces and deploy it to Databricks Model Serving in 5 steps, following the canonical OpenAI MCP Tool Calling Agent notebook verbatim. Produces a working endpoint testable in AI Playground and consumable by 06-appkit-serving-wiring. No evaluation, memory, or prompt registry — add those later via the existing worker skills. Use when creating a simple agent, scaffolding a new agent, building a quick tool-calling agent, or connecting an agent t...

End-to-end orchestrator for designing complete Gold layer schemas with ERDs, YAML files, lineage tracking, and comprehensive business documentation. Guides users through dimensional modeling, ERD creation (master/domain/summary based on table count), YAML schema generation, column-level lineage documentation, business onboarding guide creation, source table mapping, and design validation. Orchestrates design-workers (05-erd-diagrams, 06-table-documentation, 01-grain-definition, 07-design-vali...

End-to-end orchestrator for implementing Gold layer tables, merge scripts, FK constraints, and Asset Bundle jobs from YAML schema definitions. Guides users through Silver contract validation, YAML-driven table creation, Silver-to-Gold MERGE operations (SCD Type 1/2 dimensions, aggregated/transaction facts, accumulating snapshots, factless facts, periodic snapshots, junk dimensions), foreign key constraint application, Asset Bundle job configuration, and post-deployment validation. Orchestrate...

Grain definition patterns for fact tables during the Gold layer design phase. Use when choosing grain types (transaction, aggregated, snapshot), documenting grain in YAML schemas, and applying the PK-grain decision tree to determine whether a fact table needs aggregation. Prevents costly table rewrites by catching grain ambiguity before implementation.

Dimension design patterns for Gold layer modeling. Covers role-playing dimensions, degenerate dimensions, junk dimensions, mini-dimensions, outrigger dimensions, denormalized hierarchies, null handling, and flags as textual attributes. Use when designing dimension tables, choosing between dimension patterns, handling NULLs in dimensions, or flattening hierarchies. Triggers on "dimension pattern", "role-playing", "junk dimension", "degenerate dimension", "mini-dimension", "hierarchy", "NULL ha...

Advanced fact table design patterns for Gold layer modeling. Covers measure additivity classification (additive, semi-additive, non-additive), factless fact tables, accumulating snapshot facts, consolidated fact tables, header/line fact patterns, late-arriving facts and dimensions, and NULL handling in measures. Use when designing fact tables beyond basic transaction/aggregate patterns, classifying measure types, or handling complex fact scenarios. Triggers on "fact pattern", "factless", "acc...

Enterprise integration patterns for Gold layer dimensional models. Covers conformed dimensions, the enterprise data warehouse bus matrix, shrunken/rollup dimensions, conformed facts, and drill-across query patterns. Use when planning dimensions shared across multiple fact tables, creating a bus matrix for enterprise integration, designing rollup dimensions, or enabling cross-process analytics. Triggers on "conformed dimension", "bus matrix", "drill-across", "shrunken dimension", "rollup", "en...

Patterns for creating clean, professional Mermaid ERD diagrams for data modeling documentation. Use when documenting Gold layer data models, creating master ERDs for complete models, domain-specific ERDs for focused views, or summary ERDs for large models (20+ tables). Includes organization strategies, syntax standards, relationship patterns, and cross-domain reference patterns.

Comprehensive documentation standards for Gold layer tables including naming conventions, column descriptions, and metadata requirements. Use when creating Gold layer tables, columns, or writing documentation to ensure dual-purpose descriptions that serve both business users and technical users (including LLMs like Genie). Includes YAML schema consultation patterns, surrogate key patterns, SCD Type 2 documentation, and implementation guidance for Silver table naming conventions.

Cross-validation of Gold layer design artifacts during the design phase. Use when validating that YAML schemas, ERDs, lineage CSVs, and PK/FK references are internally consistent before handing off to implementation. Catches design-time inconsistencies (e.g., column in ERD but not in YAML, FK referencing non-existent table) that would otherwise surface as runtime bugs.

Active, source-bounded alignment of a Gold dimensional design to Databricks Industry Vibe Data Models (and other canonical industry reference models — TM Forum SID, ARTS, ACORD, HL7, BIAN). Use when the customer operates in a recognizable vertical (retail, banking, healthcare, telecom, insurance, hospitality) and wants the Gold layer to cover industry-standard entities and use industry terminology, when checking industry coverage/gaps, or when the customer has a Vibe-generated Silver business...

Patterns for creating Gold layer tables dynamically from YAML schema definitions at runtime. Use when managing 10+ Gold layer tables across multiple domains, when schema evolves frequently, or when you want to avoid embedded SQL DDL strings in Python. Includes YAML schema structure, setup script implementation, Asset Bundle configuration, and workflow patterns for schema changes.

Provides production-grade patterns for Gold layer MERGE operations from Silver to Gold tables. Covers column mapping, schema evolution, SCD Type 1/2 patterns, fact table aggregation, and preventing variable naming conflicts with PySpark functions. Use when creating Gold layer MERGE operations, handling column name differences between Silver and Gold, implementing SCD Type 1/2 dimensions, aggregating fact tables, or troubleshooting MERGE errors. Triggers on "Gold merge", "MERGE operation", "up...

Delta MERGE deduplication pattern for Gold layer to prevent duplicate source key errors. Use when merging from Silver to Gold tables where Silver may contain duplicate business keys due to incremental DLT streaming, CDC patterns, SCD Type 2 tracking, or multiple batch loads. Prevents DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE errors by deduplicating before merge.

Pre-merge grain validation patterns for Gold layer fact tables. Use when creating fact table merge scripts to validate that DataFrame grain matches DDL PRIMARY KEY, prevent transaction vs aggregated confusion, and catch grain mismatches before MERGE. Includes grain inference functions, pre-merge validation, validation SQL, and common grain mismatch errors.

Runtime schema validation patterns for Gold layer merge scripts. Use when creating merge scripts to ensure DataFrame columns match target DDL schemas, validate column mappings before MERGE operations, and catch schema issues before deployment. Includes the DDL-first workflow, validate_merge_schema() helper, explicit column mapping, DDL schema reader, and pre-deployment validation script. Addresses the "three sources of truth" problem by enforcing DDL as runtime truth.

MLflow and ML Model patterns for Databricks including experiment creation, model training, batch inference, and Unity Catalog integration. Use when implementing ML pipelines, training models with Feature Store, or deploying batch inference jobs. Includes 19 non-negotiable rules covering experiment paths, dataset logging, UC model registration, NaN handling, label binarization, feature engineering workflows, and signature-driven preprocessing.

End-to-end orchestrator for setting up Databricks observability including Lakehouse Monitoring, Anomaly Detection, AI/BI Dashboards, and SQL Alerts. Guides users through monitor creation for Gold tables, schema-level anomaly detection for freshness/completeness, dashboard design with monitoring widgets, and config-driven alerting. Orchestrates mandatory dependencies on monitoring skills (lakehouse-monitoring-comprehensive, anomaly-detection, databricks-aibi-dashboards, sql-alerting-patterns) ...

Comprehensive guide for Databricks Lakehouse Monitoring (Data Profiling) with quick-start workflow (2 hours), fill-in-the-blank requirements template, concrete fact/dimension monitor examples, and complete deployment patterns. Uses the new Data Quality API (`databricks.sdk.service.dataquality`). Use when setting up Lakehouse Monitoring for Gold layer tables, creating custom business metrics, designing monitoring strategy, querying monitoring tables for dashboards, or troubleshooting monitor i...

Production-grade patterns for Databricks AI/BI (Lakeview) dashboards. Prevents visualization errors, deployment failures, and maintenance issues through widget-query alignment, number formatting, parameter configuration, monitoring table patterns, chart scale properties, and automated deployment workflows. Includes pivot tables with hierarchy drill-down and ratio metrics, point/choropleth maps, sankey diagrams, waterfall and histogram charts, cross-filtering and drill-through patterns, filter...

Comprehensive guide for Databricks SQL Alerts V2 - config-driven alerting framework with SDK deployment, hierarchical job architecture (5 atomic + 1 composite), proactive EXPLAIN-based query validation, and partial success patterns. Use when setting up SQL alerts, creating alert configuration tables, deploying alerts via Databricks SDK (V2 dict-based or typed classes), or troubleshooting alert failures. Includes config-driven patterns, fully qualified table names (no parameters), severity-bas...

Schema-level anomaly detection for Databricks Unity Catalog using the Data Quality API (Public Preview). Automatically monitors table freshness and completeness using ML models. Use when setting up schema-wide data reliability monitoring, detecting stale or incomplete tables, configuring anomaly detection alerts, or querying the system results table. **Auto-triggered by Silver and Gold layer setup workflows** to ensure every new schema has baseline freshness/completeness monitoring from day one.

Create multi-phase project plans for Databricks data platform solutions with Agent Domain Framework and Agent Layer Architecture. Includes interactive Quick Start with key decisions, industry-specific domain patterns, complete phase document templates (Use Cases, Agents, Frontend), Genie Space integration patterns, deployment order requirements, and worked examples. Supports both acceleration mode (plan on a completed Gold layer) and workshop mode (`planning_mode: workshop`) that plans from t...

End-to-end orchestrator for building the Databricks semantic layer including Metric Views, Table-Valued Functions (TVFs), and Genie Spaces. Guides users through metric view creation, TVF development, Genie Space setup, and API-driven deployment. Orchestrates mandatory dependencies on semantic-layer skills (metric-views-patterns, databricks-table-valued-functions, genie-space-patterns, genie-space-export-import-api) and common skills (databricks-asset-bundles, databricks-expert-agent, databric...

Standard patterns for creating Databricks Metric Views with semantic metadata for Genie and AI/BI. Use when creating metric views, troubleshooting metric view creation errors, validating schema references before deployment, implementing joins (including snowflake schema patterns), or optimizing metric views for Genie natural language queries.

End-to-end guide for planning, creating, deploying, and validating Table-Valued Functions (TVFs) in Databricks optimized for Genie Space natural language queries. Use when creating TVFs for Genie Spaces, planning TVF requirements from business questions, troubleshooting TVF compilation errors, or ensuring Genie compatibility. Includes requirements gathering templates, schema validation patterns, SQL requirements (STRING parameters, parameter ordering, LIMIT workarounds), v3.0 bullet-point com...