Batch Integrations

---
name: batch-integrations
description: "Mage batch data integration pipelines, io_config.yaml, and
  connector configuration."
metadata:
  disco-role: operating
disable-model-invocation: true
license: Apache 2.0
---

# Batch Integrations

Use this route for non-streaming data integration pipelines and connector configuration.

## Owns

- `io_config.yaml` layout and profile resolution
- Batch data integration pipelines that move data between one source and one destination
- Source/destination configuration, schema settings, bookmarks, and stream selection
- Variable interpolation in integration configs
- Connector families exposed by the repo and the sibling `mage_integrations` package
- Safe validation of connector configuration without running the connector against a live service

## Excludes

- Batch block code itself, which belongs in `pipeline-authoring`
- Real-time message streams, which belong in `streaming`
- dbt orchestration, which belongs in `dbt-workflows`
- CLI bootstrap, auth, logging, and deployment, which belong in `platform-ops`
- AI generation, which belongs in `ai-workflows`

## Use this route when

- The user asks how to configure a source or destination profile
- The user wants to understand `io_config.yaml`
- The user needs help with data integration pipeline schema, bookmark, or prefix behavior
- The user wants to extend or reason about connector configuration

## Bundled reference

Read `references/connectors-and-config.md` for the config shape, interpolation rules, connector families, and source/destination workflow.

## Bundled troubleshooting

Read `references/troubleshooting.md` for Docker, profile, credential, local-host, and S3 credential failures.

## Bundled script

- `scripts/inspect_io_config.py` — safely inspect profiles and key presence without printing secrets

## Core reminders

- The docs and source treat data integrations as Docker-oriented because of connector dependencies.
- `io_config.yaml` belongs in the project root.
- Use this route for the configuration layer; use `pipeline-authoring` for the block logic that consumes the integration output.