Datasources And Assets

---
name: datasources-and-assets
description: "Connect GX Core data sources, data assets, batch definitions, and
  optional backends."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Datasources and Assets

Use this sub-skill when a task needs to connect GX Core to data before suites, validations, or checkpoints can run. It covers fluent datasource factories, dataframe/file/SQL assets, batch definitions, batch parameters, and optional backend dependency troubleshooting.

## Start Here

1. Create or retrieve a context first; read `../contexts-and-configuration/SKILL.md` when the task involves file projects, ephemeral contexts, config variables, or Cloud mode.
2. Choose a datasource family and asset type using `references/fluent-datasource-api.md`.
3. Add a batch definition and learn the required `batch_parameters` from `references/batch-definitions.md`.
4. If a backend import, credential, connection, or storage error appears, read `references/optional-backends.md` and `references/troubleshooting.md` before changing code.

## Bundled References and Scripts

- Read `references/fluent-datasource-api.md` to choose `context.data_sources` CRUD methods, fluent datasource factories, pandas/filesystem/SQLite/SQL assets, and table/query patterns.
- Read `references/batch-definitions.md` to define dataframe, file, directory, SQL table/query batches and diagnose required batch parameter keys.
- Read `references/optional-backends.md` before using Spark, cloud storage, warehouses, SQLAlchemy drivers, credentials, or backend-specific extras.
- Read `references/troubleshooting.md` when datasource creation, asset discovery, batch lookup, pandas reader options, SQL queries, or credentials fail.
- Run `scripts/smoke_datasource_asset.py --help` to inspect a safe local smoke helper; run it to prove that an ephemeral context can create pandas dataframe and local CSV filesystem assets without network or credentials.

## Routing Boundaries

- For context selection, config variables, and project persistence, use `../contexts-and-configuration/SKILL.md`.
- For expectation classes, suites, row conditions, and custom expectations, use `../expectations-and-suites/SKILL.md` after a batch is available.
- For running validations, passing `batch_parameters`, result formats, and unexpected rows, use `../validations-and-results/SKILL.md`.
- For checkpoints, actions, notifications, and Data Docs, use `../checkpoints-actions-and-data-docs/SKILL.md` after validations are defined.

## Default Workflow

1. Select the smallest backend that matches the data: `add_pandas` for in-memory dataframes, `add_pandas_filesystem` for local/network files, `add_sqlite` for SQLite, or `add_sql`/warehouse factories for other SQL engines.
2. Add an asset: dataframe assets use `add_dataframe_asset`; filesystem assets use file reader assets such as `add_csv_asset`, `add_excel_asset`, or `add_parquet_asset`; SQL assets use `add_table_asset` or `add_query_asset`.
3. Add a batch definition on the asset, not on the context. Use whole-data definitions for one batch, regex or column partition helpers for recurring batches, and stable names for validation reuse.
4. Inspect `asset.get_batch_parameters_keys(partitioner=batch_definition.partitioner)` and call `batch_definition.get_batch(batch_parameters=...)` with only those keys.
5. Hand the resulting `BatchDefinition` to validation code rather than embedding datasource setup inside suites or checkpoints.