Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
Installs into .claude/skills of the current project.
Are you the author of Spark Optimization?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/dokhacgiakhoa-spark-optimization)
---
version: 4.1.0-fractal
name: spark-optimization
description: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
---
# Apache Spark Optimization
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
## Do not use this skill when
- The task is unrelated to apache spark optimization
- You need a different domain or tool outside this scope
## Instructions
- Clarify goals, constraints, and required inputs.
- Apply relevant best practices and validate outcomes.
- Provide actionable steps and verification.
- If detailed examples are required, open `resources/implementation-playbook.md`.
## Use this skill when
- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
- Debugging Spark performance issues
- Scaling Spark pipelines for large datasets
- Reducing shuffle and data skew
## Core Concepts
## 🧠 Knowledge Modules (Fractal Skills)
### 1. [1. Spark Execution Model](./sub-skills/1-spark-execution-model.md)
### 2. [2. Key Performance Factors](./sub-skills/2-key-performance-factors.md)
### 3. [Pattern 1: Optimal Partitioning](./sub-skills/pattern-1-optimal-partitioning.md)
### 4. [Pattern 2: Join Optimization](./sub-skills/pattern-2-join-optimization.md)
### 5. [Pattern 3: Caching and Persistence](./sub-skills/pattern-3-caching-and-persistence.md)
### 6. [Pattern 4: Memory Tuning](./sub-skills/pattern-4-memory-tuning.md)
### 7. [Pattern 5: Shuffle Optimization](./sub-skills/pattern-5-shuffle-optimization.md)
### 8. [Pattern 6: Data Format Optimization](./sub-skills/pattern-6-data-format-optimization.md)
### 9. [Pattern 7: Monitoring and Debugging](./sub-skills/pattern-7-monitoring-and-debugging.md)
### 10. [Do's](./sub-skills/dos.md)
### 11. [Don'ts](./sub-skills/donts.md)
