Claude Skills by databricks-solutions

Patterns for setting up Databricks Genie Spaces with comprehensive agent instructions, data assets, SQL expressions, and benchmark questions. Use when creating Genie Spaces, configuring agent behavior, selecting data assets, defining SQL expressions (measures, filters, dimensions), or validating benchmark questions. Includes mandatory 8-section deliverable structure, General Instructions (≤20 lines), data asset organization (Metric Views → TVFs → Tables), SQL expressions (sql_snippets) for st...

Comprehensive patterns for Databricks Genie Space Export/Import API - JSON schema, serialization format, and programmatic deployment. Use when programmatically creating, exporting, or importing Genie Spaces via REST API, troubleshooting API deployment errors, or implementing CI/CD for Genie Spaces. Includes complete GenieSpaceExport schema, API endpoints (List, Get, Create, Update, Delete), JSON format requirements, ID generation, variable substitution, inventory-driven generation patterns, a...

End-to-end orchestrator for creating Silver layer pipelines using Spark Declarative Pipelines (SDP, formerly DLT) with Delta table-based data quality rules, quarantine patterns, and monitoring views. Orchestrates mandatory dependencies on common skills (databricks-table-properties, databricks-python-imports, databricks-asset-bundles, schema-management-patterns, unity-catalog-constraints, databricks-expert-agent) and Silver-domain skills (dlt-expectations-patterns, dqx-patterns). Use when crea...

Spark Declarative Pipeline (SDP, formerly DLT) expectations patterns for data quality with Unity Catalog Delta table storage. Use when implementing Silver layer SDP/DLT pipelines, creating portable data quality rules, or needing runtime-updateable expectations without code deployment. Supports severity-based filtering (critical vs warning) and quarantine patterns. Standardizes on `import dlt` for the DQ-rules framework; the modern `dp` API (`from pyspark import pipelines as dp`) also supports...

Databricks DQX framework patterns for advanced data quality validation with detailed failure insights and flexible quarantine strategies. Use when implementing Silver/Gold layer validation, needing richer diagnostics than DLT expectations, or requiring pre-merge validation with detailed failure tracking. Supports YAML configuration, Delta table storage, and serverless compute compatibility.

Intelligent skill navigation system with tiered loading and orchestrator-first routing for context-efficient agent operation. Routes tasks to the correct domain skill based on keyword detection with orchestrator priority. Each skill uses progressive disclosure with references/, scripts/, and assets/ directories. Use this skill as the entry point for any Databricks-related task to determine which specialized skills to load.

GenAI agent skill navigator for the `genai-agents/` folder. Start here to route agent-building, MLflow GenAI, tracing, Databricks Apps Track A, SDLC, evaluation, deployment, monitoring, prompt iteration, and vibecoding runtime tasks to the correct specialized skill. Uses tiered loading and keyword routing so agents read only the skills needed for the current task.

Use when any agent or downstream skill needs Unity Catalog schemas and managed volumes (knowledge_sources, agent_outputs, memory tables, benchmark tables, etc.) provisioned idempotently. Foundation Step 0 — runs before MLflow tracing setup and before any track-specific skill. Owned by no track. Reads the resolved spec to discover which volumes a use case needs and creates them alongside the canonical agent + ops UC schemas.

Use when deriving a Databricks Agent Spec and Agent Tool Plan from docs/design_prd.md before building Track A agents. Covers web-researched MCP recommendations, Databricks managed MCP choices, dynamic SQL MCP catalog/schema inputs, optional Knowledge Assistant selection, and validation of docs/agent_spec.yaml plus docs/agent_tool_plan.yaml.

Use when starting any new GenAI agent project on Databricks. Covers MLflow 3.x GenAI setup: install dependencies, enable autolog, configure connection pooling, understand model signatures, detect runtime environment. Foundation Step 1. Produces the configured MLflow environment consumed by all subsequent skills.

Use when setting up MLflow experiments, tracing, or UC OTEL trace storage for a GenAI agent. Covers structured experiment paths, tracing decorators, manual spans, tags, connection pooling, and Unity Catalog OTEL storage for SQL-queryable trace retention. Foundation Step 2. Consumes MLflow environment from Step 1.

Use when adding MLflow tracing to a Node/TypeScript GenAI agent. Covers the official `mlflow-tracing` and `mlflow-openai` npm packages: `mlflow.init()`, `tracedOpenAI`, manual `mlflow.trace`/`@mlflow.trace`/`mlflow.withSpan`, nested spans, sessions/users, and when to choose a custom OTLP fallback. Foundation Step 2b. Sibling to the Python tracing skill (Foundation Step 2). Consumes the MLflow environment from Step 1.

Use when adding user, session, environment, or version context to MLflow traces — Python or TypeScript. Covers the reserved metadata fields (`mlflow.trace.user`, `mlflow.trace.session`), auto-populated environment metadata (`mlflow.source.*`, Git provenance, model id), the `APP_ENVIRONMENT` override pattern for production, custom deployment metadata, and how this composes with `client_request_id` and `gen_ai.*`. Foundation Step 2c. Sibling to F2 (Python tracing) and F2b (TS tracing).

Use when connecting any agent to data and external capabilities. Covers Databricks managed MCP servers (Vector Search, Genie, SQL, UC Functions), the databricks-mcp client library, _meta parameters, external MCP servers, code interpreter, non-MCP connections, resource grants, custom retriever schemas, tool cookbook (web search, SQL, file generation, HTTP), runtime guardrails, and end-to-end external MCP examples. Foundation Step 3. Consumed by all agent tracks (A, B, C).

Use when you need central governance, observability, guardrails, or rate limits across LLM endpoints and MCP servers used by agents. Covers creating and configuring Databricks AI Gateway endpoints, enabling usage tracking, inference tables (UC Delta audit), rate limits, and guardrails, plus how deployed agent serving endpoints and Apps consume a gateway URL. Foundation Step 4 — ideal as a governance layer BEFORE Module 7 production deployment. Complements in-code guardrails and MLflow tracing...

Use when creating, syncing, or managing a Databricks Agent Bricks Knowledge Assistant (KA) as a document Q&A source for an agent. Covers source mode selection (UC Files vs Vector Search index), Python SDK lifecycle (create/update/sync/delete), REST API equivalents, readiness polling, and emitting the ka_endpoint_name handoff that downstream agent skills wire as a knowledge_assistant tool. Track-neutral foundation skill. Optional but recommended whenever document Q&A is part of the agent.

Use when you need to version, manage, or load prompts for any GenAI agent on Databricks. Covers how to register prompts in Unity Catalog, set up aliases for safe deployments, link prompts to traces, and load them in production apps — even if you just want "register my prompt and give me a stable URI." Also use when A/B testing prompt versions or connecting prompt changes to evaluation runs. SDLC Step 1.

Use when you need to create, manage, or load evaluation datasets for testing agent quality. Covers the MLflow GenAI data format, persisting benchmarks in Unity Catalog, merging records without duplicates, and validating data before evaluation — even if you just want "give me a dataset I can pass to mlflow.genai.evaluate()." Also use when building benchmarks from production traces or SME labels. SDLC Step 2.

Use when you need to measure agent quality or create scoring criteria for evaluation gates. Covers how to pick and configure scorers — even if you just want "is my agent safe and accurate?" without knowing which MLflow classes to use. Also use when building custom LLM judges, evaluating multi-turn conversations, or setting pass/fail thresholds. SDLC Step 3.

Use when running mlflow.genai.evaluate() to test agent quality before deployment. Covers the predict_fn contract, answer-sheet mode for re-scoring existing outputs, threshold gates, retry wrappers, human feedback sessions, and conversation evaluation — even if you just want "run my benchmarks and tell me pass or fail." Also use when collecting human labels to calibrate automated scorers. SDLC Step 4.

Use after evaluation (Step 4) passes but BEFORE deployment (Step 6). Covers translating technical eval metrics to business-meaningful terms, running a structured review with business and compliance stakeholders, capturing sign-off, and blocking promotion if review-blocking issues remain. Mirrors the "Align with stakeholders before production" phase of the Databricks agents development workflow. SDLC Step 4b. Do NOT use this skill in place of Step 4 eval — this is the gate layered on top of it.

Use when wiring **end-user feedback** (thumbs up/down, ratings, free-form comments) from a deployed app back into MLflow as Assessments on the originating trace. Covers the canonical `mlflow.log_feedback(...)` pattern, trace-id vs `client_request_id` correlation, the streaming case (SSE/WebSocket delivery of `trace_id` after the stream ends), multi-dimensional feedback (rating-per-aspect), update/delete of existing assessments, and analysis queries over collected feedback. Triggers on "collec...

Use when you need to log an agent to MLflow, register it in Unity Catalog, or promote a new version as champion. Covers the full model lifecycle — even if you just want "log my agent and make it the production version." Also use when setting up MLFLOW_ACTIVE_MODEL_ID for production trace linking, metric-based promotion gating, or rollback. SDLC Step 5.

Use when deploying an agent to production or setting up CI/CD automation. Covers Databricks Apps deployment, Asset Bundles, service principal permissions, and evaluate-then-promote pipelines — even if you just want "deploy my agent and set up a release gate." Also use for MCP tool connectivity, Supervisor long-running tasks, or production trace linking. SDLC Step 6.

Use when setting up live quality monitoring for a deployed agent. Covers registered scorers with sampling, trace archival to Delta, metric backfill, and periodic human review — even if you just want "monitor my agent's safety score in production." Also use when adding conversation-level judges, tuning sampling rates, or building monitoring dashboards from UC trace data. SDLC Step 7.

Use when prompt quality has plateaued on your evaluation dataset and you need to programmatically improve one or more registered prompts. Covers `mlflow.genai.optimize_prompts()` and `GepaPromptOptimizer`, `predict_fn` contracts, single and multi-prompt optimization, custom scorers, budget controls, registering the optimized output back to UC with a staging alias, and the promote-after-eval loop. SDLC Step 8 — run AFTER Step 4 evaluation reveals specific regressions or unmet guideline checks....

Default reusable MLflow workshop path for instruction-shaped prompt iteration. Hand-authored prompt revisions guided by failing-scorer rationales from the first scored eval, with full-dataset re-eval and alias-gated promotion. Use when Step 4 evaluation reveals specific prompt-related failure modes (guideline misses, missing structure, wrong tone) AND failure_shape is `instruction` AND there are no L1 scorer failures. Cheaper, more transparent, and lower-risk than automated optimization. SDLC...

Use when starting a new custom agent project on Databricks Apps. Clone the agent-openai-agents-sdk template, run quickstart, start the local dev server, verify the built-in chat UI. Track A Step 1. Produces a running local agent project ready for customization.

Use when building a custom agent with the OpenAI Agents SDK and MLflow ResponsesAgent. Covers Agent class, Runner, @function_tool, handoffs, streaming, ModelConfig, mlflow.models.set_model(), autolog, and manual tracing. Track A Step 2. Produces a customized, MLflow-compatible agent class ready for tools and deployment.

Use when wiring tools into an OpenAI Agents SDK agent: local Python function tools, DatabricksMCPClient connections, and MCPServerSse for MCP servers. Track A Step 3. Builds on shared concepts from F3 (Tools and Data Access).

Use when configuring agent permissions, implementing OBO, or granting access to Databricks resources. Covers App Authorization (service principal), User Authorization (OBO), auth passthrough for Model Serving, and MCP server authentication. Track A Step 4.

Use when implementing stateful agents that need conversation memory. Covers short-term continuity via AsyncDatabricksSession (prescribed for OpenAI Agents SDK), long-term preferences via DatabricksStore, graceful degradation, and thread_id resolution. Track A Step 5. Consumes authenticated client from Step 4.

Use when evaluating an agent's response quality and safety before deployment. Covers running agent-evaluate, evaluation dataset format, built-in judges (relevance, groundedness, safety), interpreting results, and customizing eval datasets. Track A Step 6. Consumes a working agent with tools from Steps 1-5. Produces evaluation results and confidence to deploy.

Use when deploying an agent to Databricks Apps and querying the deployed endpoint. Covers databricks apps create, sync, deploy, OAuth token auth, PAT limitations, curl and Python SDK query methods, and the redeploy workflow. Track A Step 7. Consumes evaluation results from Step 6. Produces a running deployed agent on Databricks Apps.

Use when a deployed Databricks Apps agent is failing, returning errors, or behaving unexpectedly. Covers systematic debugging of local dev, bundle configuration, deployment, runtime errors, authentication, resource permissions, and Lakebase memory. Track A Step 8. Consumes a deployed app from Step 7. Produces a resolved, healthy agent deployment.

Lakebase Change Data Feed (CDF) captures every `INSERT`, `UPDATE`, and `DELETE` on Lakebase Postgres tables via the write-ahead log (WAL) and stores them as append-only rows in Unity Catalog managed Delta tables. Changes are batched and flushed approximately every ~15 seconds.

Standard patterns for Databricks Asset Bundles configuration files for serverless jobs, DLT pipelines, dashboards, alerts, apps, and workflows. Use when creating, configuring, or deploying DABs for infrastructure-as-code deployments. Covers mandatory serverless environment configuration, hierarchical job architecture (atomic/composite/orchestrator), DLT pipeline patterns, dashboard resources with dataset_catalog/dataset_schema, SQL Alerts v2 API schema, Apps lifecycle, Python notebook paramet...

Transforms the assistant into a Senior Databricks Solutions Architect Agent that designs, implements, and reviews production-grade Databricks solutions following official best practices. Enforces Unity Catalog governance, Delta Medallion architecture, DLT expectations, Predictive Optimization, automatic liquid clustering, UC Metric Views, Genie TVFs, Serverless Workflows, and Asset Bundles. Use when working on Databricks projects requiring production-grade solutions with governance, quality, ...

Session-start behavioral manifest for running this workshop inside Databricks Genie Code. Read FIRST when the client is Genie Code (the coding agent embedded in the Databricks workspace) so you begin knowing how it behaves — surface/page tool-scoping, the three execution paths (runDatabricksCli → SDK → native tools) and the "blocked ≠ impossible, try the next path" discipline, the runDatabricksCli allow-list tiers, bundle-deploy reality (--target dev mandatory, CWD pinned to the page's bundle...

Owns the end-to-end state, gating, pathway-applicability, and retrospective contract for the `Instructions.md` / `example/<use_case>/WALKTHROUGH.md` vibecoding workshops. Every workshop prompt invokes this skill at the top (`enter`) and bottom (`exit`) so the state file is read, gates are enforced, non-applicable steps are skipped, and the per-step log is appended — without each prompt repeating the contract as prose. Also owns the `bootstrap` initializer, the bootstrap → canonical `migrate_c...