Synapse Pipelines

---
name: synapse-pipelines
description: "Azure Synapse Pipelines specialist for data integration within the Synapse Analytics unified workspace. Deep expertise in pipelines, Spark pool integration, dedicated/serverless SQL pools, Mapping Data Flows, Synapse Link, CI/CD, IR configuration, and migration to Microsoft Fabric. WHEN: \"Synapse Pipelines\", \"Synapse workspace\", \"Synapse pipeline\", \"Synapse Spark pool\", \"dedicated SQL pool\", \"serverless SQL pool\", \"Synapse notebook activity\", \"Synapse Studio\", \"Synapse Monitor hub\", \"Synapse Link\", \"Synapse Data Flow\", \"Synapse IR\", \"Synapse SHIR\", \"Synapse RBAC\", \"Synapse CI/CD\", \"workspace_publish\", \"Synapse vs ADF\", \"Synapse vs Fabric\", \"Synapse migration\", \"Data Exfiltration Protection\", \"Managed VNET Synapse\", \"Synapse trigger\"."
license: MIT
---

# Azure Synapse Pipelines

This skill covers Azure Synapse Pipelines, the data integration component of Azure Synapse Analytics. Synapse Pipelines share the same execution engine as Azure Data Factory but are embedded within a Synapse workspace, providing native integration with Spark pools, dedicated SQL pools, serverless SQL pools, and Synapse Link. Synapse is a managed service with no user-facing version numbers. It provides deep knowledge of:

- Pipeline orchestration (Copy Activity, control flow, 90+ connectors, Mapping Data Flows)
- Native Spark integration (Synapse Notebook activity, Spark Job Definition activity, pool configuration)
- Native SQL pool integration (dedicated SQL pool with DWU scaling, serverless SQL pool, COPY INTO, Stored Procedure)
- Integration Runtimes (Azure IR with Managed VNET, Self-Hosted IR -- no Azure-SSIS IR in Synapse)
- Synapse Link (zero-ETL for Cosmos DB, SQL, Dataverse)
- Triggers (schedule, tumbling window, storage event, custom event)
- CI/CD with Git integration and Synapse workspace deployment task
- Security (Managed VNET, Data Exfiltration Protection, Synapse RBAC, managed identity)
- Monitoring (Synapse Studio Monitor hub, Azure Monitor, Log Analytics, KQL)
- Migration path to Microsoft Fabric

## How to Approach Tasks

When you receive a request:

1. **Classify** the request:
   - **Architecture / pipeline design** -- Load `references/architecture.md` for workspace model, IR types, Spark pools, SQL pools, data flows, triggers, linked services
   - **Performance / best practices** -- Load `references/best-practices.md` for pipeline design, Spark pool sizing, SQL pool optimization, security, cost management, CI/CD
   - **Troubleshooting / diagnostics** -- Load `references/diagnostics.md` for pipeline failures, Spark errors, SQL pool issues, IR connectivity, monitoring
   - **ADF comparison** -- See the `adf` skill for ADF-specific features (global parameters, Azure-SSIS IR) not available in Synapse
   - **Fabric migration** -- Use the migration section below and `references/best-practices.md` for migration planning
   - **Cross-tool comparison** -- See the `integration` skill for Synapse vs Glue, Fivetran, NiFi, etc.

2. **Gather context** -- Determine:
   - What compute type? (Spark pool, dedicated SQL pool, serverless SQL pool, Data Flow)
   - What does the pipeline do? (data movement, Spark transformation, SQL transformation, orchestration)
   - Is the workspace using Managed VNET? Data Exfiltration Protection?
   - Is Git integration configured? What is the CI/CD approach?
   - Is there interest in migrating to Microsoft Fabric?

3. **Analyze** -- Apply Synapse-specific reasoning. Consider workspace-centric architecture, native Spark/SQL pool integration, Managed VNET implications, the absence of global parameters, and the CI/CD differences from standalone ADF.

4. **Recommend** -- Provide actionable guidance with specific Synapse configuration, T-SQL commands, Spark pool settings, KQL queries, and Azure CLI/PowerShell commands where appropriate.

5. **Verify** -- Suggest validation steps (Monitor hub inspection, Spark UI review, DMV queries, Log Analytics KQL queries).

## Core Architecture

### Workspace-Centric Model

```
┌─────────────────────────────────────────────────────┐
│  Synapse Workspace                                  │
│  ┌────────────┐  ┌────────────┐  ┌──────────────┐  │
│  │  Pipelines │  │ Spark Pools│  │  SQL Pools   │  │
│  │ (Integrate)│  │ (Develop)  │  │ (Dedicated + │  │
│  │            │  │            │  │  Serverless) │  │
│  └─────┬──────┘  └──────┬─────┘  └──────┬───────┘  │
│        │                │               │           │
│  ┌─────▼────────────────▼───────────────▼────────┐  │
│  │         Synapse Studio (Monitor Hub)          │  │
│  └───────────────────────┬───────────────────────┘  │
│                          │                          │
│  ┌───────────────────────▼───────────────────────┐  │
│  │     Integration Runtimes (Azure IR / SHIR)    │  │
│  └───────────────────────┬───────────────────────┘  │
│                          │                          │
│  ┌───────────────────────▼───────────────────────┐  │
│  │        ADLS Gen2 (Primary Storage)            │  │
│  └───────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────┘
```

**Synapse Studio** provides a unified web-based IDE for data integration, data exploration, warehousing, big data analytics, and ML -- a single pane of glass for data engineers, analysts, and scientists.

**Primary storage**: Every workspace is associated with an ADLS Gen2 account. This is the default data lake for Spark pools, serverless SQL, and pipeline staging.

### Synapse Pipelines vs ADF -- Key Differences

| Aspect | ADF | Synapse Pipelines |
|---|---|---|
| **Deployment model** | Standalone service | Embedded within Synapse workspace |
| **SQL pool integration** | External linked service only | Native with dedicated and serverless pools |
| **Spark integration** | HDInsight or Databricks (external) | Native Spark pools, notebook activity |
| **Global parameters** | Supported | **Not supported** -- significant gap |
| **Azure-SSIS IR** | Fully supported | **Not supported** -- use standalone ADF |
| **CI/CD** | ARM template deployment task | Synapse workspace deployment task (artifacts are not ARM resources) |
| **Workspace DB source** | Not available | Data flows reference workspace DBs directly |
| **Strategic direction** | Stable, fully supported | Stable, but new features increasingly Fabric-only |

**Shared foundation**: Same pipeline execution engine, same activity runtime, same 90+ connectors, same data flow engine, same trigger types, same Self-Hosted IR.

### Integration Runtimes

| IR Type | Compute | Network | Notes |
|---|---|---|---|
| **Azure IR** | Serverless, auto-scaling | Public or Managed VNET with private endpoints | Auto-resolve or fixed-region |
| **Self-Hosted IR** | Windows machine or VM | Outbound HTTPS (443) only | Multi-node HA, shared across workspaces |
| **Azure-SSIS IR** | -- | -- | **Not supported in Synapse** -- use ADF |

**Managed VNET**: When enabled, all Synapse compute runs inside a managed virtual network. Managed private endpoints connect to Azure services via Private Link. **This is a permanent decision set at workspace creation and cannot be changed later.**

**Data Exfiltration Protection (DEP)**: Restricts all outbound traffic to approved Microsoft Entra tenants only. Prevents unauthorized data transfers. **Also permanent once enabled at workspace creation.**

### Apache Spark Pools

Spark pools provide serverless Apache Spark compute within the workspace:

| Node Size | vCores | Memory |
|---|---|---|
| Small | 4 | 32 GB |
| Medium | 8 | 64 GB |
| Large | 16 | 128 GB |
| XLarge | 32 | 256 GB |
| XXLarge | 64 | 432 GB |

**Configuration**: Auto-scale (3-200 nodes), auto-pause (5 min to 7 days), dynamic executor allocation, TTL for warm clusters, Spark version pinning, library management at pool or session level.

**Pipeline integration**: Synapse Notebook activity and Spark Job Definition activity execute on Spark pools with pipeline parameter passing and output capture.

### SQL Pools

**Dedicated SQL Pool** (formerly SQL DW): Provisioned MPP data warehouse. Capacity measured in DWUs. Data distributed across 60 distributions (hash, round-robin, replicated). Supports T-SQL with MPP extensions (CTAS, distribution hints, result set caching). Can be paused/resumed.

**Serverless SQL Pool**: On-demand, pay-per-query SQL engine. Queries data in-place from ADLS Gen2, Cosmos DB (via Synapse Link), and external sources using OPENROWSET and external tables. Every workspace gets one built-in serverless pool.

### Mapping Data Flows

Visual, code-free Spark-based transformations within Synapse Studio:

- Same engine as ADF data flows
- **Workspace DB source**: Unique to Synapse -- reference workspace databases directly without linked services
- Schema drift handling, debug mode with live data preview
- Flowlets for reusable transformation logic
- External Call transformation for REST API integration

### Triggers

| Type | Behavior | State |
|---|---|---|
| **Schedule** | Fires at wall-clock intervals | Fire-and-forget; does not track pipeline outcome |
| **Tumbling Window** | Fixed-size intervals from start time | Stateful; retains state, supports retry and dependencies |
| **Storage Event** | Blob created/deleted (Event Grid) | Event-driven; filters by path prefix/suffix |
| **Custom Event** | Event Grid custom topics | Event-driven; filters by event type and subject |

**Critical behavior**: Triggers must be explicitly **published** and **started** -- they are not active in draft/unpublished state. This is a common source of "pipeline not running" issues.

### Synapse Link

Zero-ETL data synchronization from operational stores into the workspace:

**Synapse Link for Cosmos DB**: Creates a column-oriented analytical store alongside the row-oriented transactional store. Queryable via Spark and serverless SQL. **No longer recommended for new projects** -- use Azure Cosmos DB Mirroring for Fabric instead.

**Synapse Link for SQL**: Near-real-time replication from Azure SQL Database or SQL Server 2022 to dedicated SQL pool via change feed.

**Synapse Link for Dataverse**: Synchronizes Dynamics 365 / Power Apps data into ADLS Gen2 (Delta/CSV). **Transitioning to Fabric Link for Dataverse.**

### Linked Services

Linked services are **workspace-scoped** -- available to all pipelines, data flows, notebooks, and SQL scripts within the workspace.

For data within the workspace (dedicated SQL pool, serverless SQL pool, Spark databases), data flows can use the **Workspace DB** source type, bypassing linked service overhead.

## Current Status and Fabric Migration

### Platform Status (April 2026)

- **Not retired**: Azure Synapse Analytics continues to be fully supported with no announced end-of-life date
- **Reduced new investment**: Most new R&D has shifted to Microsoft Fabric
- **Retired component**: Synapse Data Explorer (Preview) retired October 7, 2025
- **All other components active**: Pipelines, Spark pools, SQL pools remain fully supported
- **Fabric trajectory**: Fabric surpassed $2B ARR with 31,000+ customers, growing 60% YoY

### Migration Assistants

Microsoft provides guided migration tools:

- **Pipeline Migration Assistant** (Preview): Accessible from Integrate hub > "Migrate to Fabric". Three stages: Assessment, Review, Migration. Auto-converts linked services to Fabric connections. Disables triggers by default after migration.
- **Spark Migration Assistant** (Preview): Migrates notebooks, Spark job definitions, and configurations.
- **Data Warehouse Migration Assistant**: Migrates from dedicated SQL pool to Fabric Data Warehouse with AI-powered T-SQL compatibility adjustments.

### Fabric Equivalents

| Synapse Component | Fabric Equivalent |
|---|---|
| Synapse Pipelines | Fabric Data Factory pipelines |
| Dedicated SQL pool | Fabric Data Warehouse |
| Serverless SQL pool | Fabric SQL Analytics Endpoint |
| Spark pools | Fabric Data Engineering (Spark) |
| Data Explorer pools | Fabric Real-Time Intelligence |
| Mapping Data Flows | Fabric Dataflows Gen2 |
| Synapse Link | Fabric Mirroring |

### Migration Guidance

1. **Assess**: Run the pipeline migration assistant to identify compatibility and unsupported activities
2. **Plan**: Map components to Fabric equivalents; model cost differences (Synapse pay-per-use vs Fabric capacity units)
3. **Pilot**: Migrate non-critical workloads first
4. **Validate**: Triggers disabled post-migration -- validate behavior before re-enabling
5. **No urgency**: Synapse is not being retired -- plan strategically, not reactively

## Anti-Patterns

1. **Using Synapse for SSIS package execution** -- Synapse does not support Azure-SSIS IR. Use standalone Azure Data Factory for SSIS lift-and-shift.
2. **Relying on global parameters** -- Synapse does not support global parameters. Use pipeline parameters and Key Vault references for environment-specific configuration.
3. **Not enabling Managed VNET at workspace creation** -- This is a permanent, irreversible decision. Evaluate security requirements before creating the workspace.
4. **Leaving dedicated SQL pools running when idle** -- Dedicated SQL pools charge DWU-hours even when no queries are running. Automate pause/resume with Azure Automation or Logic Apps.
5. **Disabling auto-pause on development Spark pools** -- Idle Spark clusters burn vCore-hours. Always enable auto-pause for dev/test pools with short timeouts (15-30 min).
6. **Making manual changes to test/prod workspaces** -- All changes must flow through Git and the CI/CD pipeline. Manual edits cause drift and deployment failures.
7. **Using ARM deployment task for Synapse artifacts** -- Synapse artifacts are not ARM resources. Use the Synapse workspace deployment task (`Synapse workspace deployment@2`).
8. **Ignoring Fabric migration assessment** -- Even if migration is not imminent, run the assessment periodically to understand compatibility and plan the eventual transition.

## Reference Files

- `references/architecture.md` -- Workspace architecture, Synapse vs ADF comparison, IR types, Spark pools (sizing, auto-scale, auto-pause), SQL pools (dedicated and serverless), data flows, triggers, linked services, Synapse Link
- `references/best-practices.md` -- Pipeline design patterns, Spark pool sizing, SQL pool optimization, security (Managed VNET, DEP, RBAC), cost management, CI/CD with Git
- `references/diagnostics.md` -- Pipeline failure patterns, Spark job errors (OOM, throttling, library conflicts), SQL pool issues (TempDB, data skew, concurrency), IR connectivity, monitoring (KQL queries, alerting)

## Cross-References

- `adf` skill -- Azure Data Factory for features not in Synapse (global parameters, Azure-SSIS IR, broader CI/CD tooling)
- `ssis` skill -- SSIS context when SSIS migration requires ADF Azure-SSIS IR
- `spark` skill -- Apache Spark context for Spark-specific tuning
- `overview` skill -- Cross-tool comparisons and paradigm routing