Spark Optimization

---
version: 4.1.0-fractal
name: spark-optimization
description: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
---

# Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

## Do not use this skill when

- The task is unrelated to apache spark optimization
- You need a different domain or tool outside this scope

## Instructions

- Clarify goals, constraints, and required inputs.
- Apply relevant best practices and validate outcomes.
- Provide actionable steps and verification.
- If detailed examples are required, open `resources/implementation-playbook.md`.

## Use this skill when

- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
- Debugging Spark performance issues
- Scaling Spark pipelines for large datasets
- Reducing shuffle and data skew

## Core Concepts

## 🧠 Knowledge Modules (Fractal Skills)

### 1. [1. Spark Execution Model](./sub-skills/1-spark-execution-model.md)
### 2. [2. Key Performance Factors](./sub-skills/2-key-performance-factors.md)
### 3. [Pattern 1: Optimal Partitioning](./sub-skills/pattern-1-optimal-partitioning.md)
### 4. [Pattern 2: Join Optimization](./sub-skills/pattern-2-join-optimization.md)
### 5. [Pattern 3: Caching and Persistence](./sub-skills/pattern-3-caching-and-persistence.md)
### 6. [Pattern 4: Memory Tuning](./sub-skills/pattern-4-memory-tuning.md)
### 7. [Pattern 5: Shuffle Optimization](./sub-skills/pattern-5-shuffle-optimization.md)
### 8. [Pattern 6: Data Format Optimization](./sub-skills/pattern-6-data-format-optimization.md)
### 9. [Pattern 7: Monitoring and Debugging](./sub-skills/pattern-7-monitoring-and-debugging.md)
### 10. [Do's](./sub-skills/dos.md)
### 11. [Don'ts](./sub-skills/donts.md)