1098 Turborepo Pipelines 64279184

# Turborepo Task Pipelines

Task orchestration, dependencies, and parallel execution strategies.

## Pipeline Configuration

Define tasks in `turbo.json`:

```json
{
  "$schema": "https://turbo.build/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**", ".next/**"]
    },
    "test": {
      "dependsOn": ["build"],
      "outputs": ["coverage/**"]
    },
    "lint": {},
    "dev": {
      "cache": false,
      "persistent": true
    }
  }
}
```

## Task Dependencies

### Topological Dependencies (^)

`^` means "run this task in dependencies first":

```json
{
  "pipeline": {
    "build": {
      "dependsOn": ["^build"]
    }
  }
}
```

Example flow:
```
packages/ui (dependency)
  ↓ builds first
apps/web (depends on @repo/ui)
  ↓ builds second
```

### Internal Dependencies

Run tasks in same package first:

```json
{
  "pipeline": {
    "deploy": {
      "dependsOn": ["build", "test"]
    }
  }
}
```

Execution order in same package:
1. Run `build`
2. Run `test`
3. Run `deploy`

### Combined Dependencies

Mix topological and internal:

```json
{
  "pipeline": {
    "test": {
      "dependsOn": ["^build", "lint"]
    }
  }
}
```

Execution order:
1. Build all dependencies (`^build`)
2. Lint current package (`lint`)
3. Run tests (`test`)

## Task Configuration Options

### outputs

Define what gets cached:

```json
{
  "build": {
    "outputs": [
      "dist/**",           // All files in dist
      ".next/**",          // Next.js build
      "!.next/cache/**",   // Exclude Next.js cache
      "build/**",          // Build directory
      "public/dist/**"     // Public assets
    ]
  }
}
```

### cache

Enable/disable caching:

```json
{
  "dev": {
    "cache": false        // Don't cache dev server
  },
  "build": {
    "cache": true         // Cache build (default)
  }
}
```

### persistent

Keep task running (for dev servers):

```json
{
  "dev": {
    "cache": false,
    "persistent": true    // Don't kill after completion
  }
}
```

### env

Environment variables affecting output:

```json
{
  "build": {
    "env": [
      "NODE_ENV",
      "NEXT_PUBLIC_API_URL",
      "DATABASE_URL"
    ]
  }
}
```

### passThroughEnv

Pass env vars without affecting cache:

```json
{
  "build": {
    "passThroughEnv": [
      "DEBUG",            // Pass through but don't invalidate cache
      "LOG_LEVEL"
    ]
  }
}
```

### inputs

Override default input detection:

```json
{
  "build": {
    "inputs": [
      "src/**/*.ts",
      "!src/**/*.test.ts", // Exclude test files
      "package.json"
    ]
  }
}
```

### outputMode

Control output display:

```json
{
  "build": {
    "outputMode": "full"        // Show all output
  },
  "dev": {
    "outputMode": "hash-only"   // Show cache hash only
  },
  "test": {
    "outputMode": "new-only"    // Show new output only
  },
  "lint": {
    "outputMode": "errors-only" // Show errors only
  }
}
```

## Running Tasks

### Basic Execution

```bash
# Run build in all packages
turbo run build

# Run multiple tasks
turbo run build test lint

# Run with specific package manager
pnpm turbo run build
```

### Filtering

Run tasks in specific packages:

```bash
# Single package
turbo run build --filter=web
turbo run build --filter=@repo/ui

# Multiple packages
turbo run build --filter=web --filter=api

# All apps
turbo run build --filter='./apps/*'

# Pattern matching
turbo run test --filter='*-api'
```

### Dependency Filtering

```bash
# Package and its dependencies
turbo run build --filter='...web'

# Package's dependencies only (exclude package itself)
turbo run build --filter='...^web'

# Package and its dependents
turbo run test --filter='ui...'

# Package's dependents only
turbo run test --filter='^ui...'
```

### Git-Based Filtering

Run only on changed packages:

```bash
# Changed since main branch
turbo run build --filter='[main]'

# Changed since HEAD~1
turbo run build --filter='[HEAD~1]'

# Changed in working directory
turbo run test --filter='...[HEAD]'

# Package and dependencies, only if changed
turbo run build --filter='...[origin/main]'
```

## Concurrency Control

### Parallel Execution (Default)

Turborepo runs tasks in parallel when safe:

```bash
# Run with default parallelism
turbo run build
```

### Limit Concurrency

```bash
# Max 3 tasks at once
turbo run build --concurrency=3

# 50% of CPU cores
turbo run build --concurrency=50%

# No parallelism (sequential)
turbo run build --concurrency=1
```

### Continue on Error

```bash
# Don't stop on first error
turbo run test --continue
```

## Task Execution Order

Example monorepo:
```
apps/
├── web (depends on @repo/ui, @repo/utils)
└── docs (depends on @repo/ui)
packages/
├── ui (depends on @repo/utils)
└── utils (no dependencies)
```

With config:
```json
{
  "pipeline": {
    "build": {
      "dependsOn": ["^build"]
    }
  }
}
```

Execution order for `turbo run build`:
1. **Wave 1** (parallel): `@repo/utils` (no dependencies)
2. **Wave 2** (parallel): `@repo/ui` (depends on utils)
3. **Wave 3** (parallel): `web` and `docs` (both depend on ui)

## Complex Pipeline Examples

### Full-Stack Application

```json
{
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": [".next/**", "dist/**"]
    },
    "test": {
      "dependsOn": ["^build"],
      "outputs": ["coverage/**"]
    },
    "lint": {
      "dependsOn": ["^build"]
    },
    "typecheck": {
      "dependsOn": ["^build"]
    },
    "dev": {
      "cache": false,
      "persistent": true
    },
    "deploy": {
      "dependsOn": ["build", "test", "lint", "typecheck"]
    }
  }
}
```

### Monorepo with Code Generation

```json
{
  "pipeline": {
    "generate": {
      "cache": false,
      "outputs": ["src/generated/**"]
    },
    "build": {
      "dependsOn": ["^build", "generate"],
      "outputs": ["dist/**"]
    },
    "test": {
      "dependsOn": ["generate"],
      "outputs": ["coverage/**"]
    }
  }
}
```

### Database-Dependent Pipeline

```json
{
  "pipeline": {
    "db:generate": {
      "cache": false
    },
    "db:migrate": {
      "cache": false
    },
    "build": {
      "dependsOn": ["^build", "db:generate"],
      "outputs": ["dist/**"]
    },
    "test:unit": {
      "dependsOn": ["build"]
    },
    "test:integration": {
      "dependsOn": ["db:migrate"],
      "cache": false
    }
  }
}
```

## Dry Run

Preview execution without running:

```bash
# See what would run
turbo run build --dry-run

# JSON output for scripts
turbo run build --dry-run=json

# Show full task graph
turbo run build --graph
```

## Force Execution

Ignore cache and run tasks:

```bash
# Force rebuild everything
turbo run build --force

# Force specific package
turbo run build --filter=web --force
```

## Output Control

```bash
# Show only errors
turbo run build --output-logs=errors-only

# Show new logs only
turbo run build --output-logs=new-only

# Show cache hash only
turbo run build --output-logs=hash-only

# Show full output
turbo run build --output-logs=full
```

## Best Practices

1. **Use topological dependencies** - `^build` ensures correct build order
2. **Cache build outputs** - Define `outputs` for faster rebuilds
3. **Disable cache for dev** - Set `cache: false` for dev servers
4. **Mark persistent tasks** - Use `persistent: true` for long-running tasks
5. **Filter strategically** - Use filters to run only affected tasks
6. **Control concurrency** - Limit parallelism for resource-intensive tasks
7. **Configure env vars** - Include vars that affect output in `env`
8. **Use dry-run** - Preview execution plan before running
9. **Continue on error in CI** - Use `--continue` to see all errors
10. **Leverage git filtering** - Run only on changed packages in CI

## Common Patterns

### CI/CD Pipeline

```yaml
# .github/workflows/ci.yml
jobs:
  build:
    steps:
      - run: turbo run build test lint --filter='...[origin/main]'
```

Only build/test/lint changed packages and their dependents.

### Development Workflow

```bash
# Start all dev servers
turbo run dev

# Start specific app with dependencies
turbo run dev --filter=web...
```

### Pre-commit Hook

```json
// package.json
{
  "scripts": {
    "pre-commit": "turbo run lint test --filter='...[HEAD]'"
  }
}
```

Only lint/test changed packages.

### Deployment

```bash
# Build and test specific app
turbo run build test --filter=web...

# Deploy if successful
turbo run deploy --filter=web
```

Build app and its dependencies, then deploy.