Spark Converter

---
name: "spark-converter"
description: "Routes Spark DataFrame conversion workflows that materialize
  cached parquet datasets and expose TensorFlow or PyTorch loaders through
  Petastorm."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Spark Converter

Use this route when the starting point is a Spark DataFrame that should be materialized once and reused many times as a TensorFlow dataset
or a PyTorch DataLoader.
This route owns the converter cache, cache cleanup, filesystem normalization, and the Spark-side API surface around
`make_spark_converter`.

## Typical triggers

- "How do I convert a Spark DataFrame into a TF dataset?"
- "How do I convert a Spark DataFrame into a PyTorch DataLoader?"
- "How do I set the converter cache directory?"
- "How do I delete converter cache files?"
- "Why does Spark converter cache cleanup or Horovod compatibility fail?"

## What belongs here

- `make_spark_converter`
- `SparkDatasetConverter`
- `make_tf_dataset`
- `make_torch_dataloader`
- converter cleanup and delete behavior
- parent cache directory configuration
- DBFS and filesystem normalization rules
- file availability waits for the converter cache
- Horovod rank/size compatibility checks

## What does not belong here

- raw dataset writing
- metadata repair
- reader-side `make_reader` and `make_batch_reader` workflows
- general Spark SQL or ML training that does not use the converter

Use `sub-skills/create-datasets/` for writing new datasets.
Use `sub-skills/read-datasets/` for consuming existing datasets directly.

## Read this first

- `references/workflows.md` for converter setup and adapter recipes
- `references/api-reference.md` for verified signatures and cache behavior
- `references/troubleshooting.md` for Spark, DBFS, cache, and optional backend failures
- `scripts/smoke_spark_converter.py` for a tiny end-to-end conversion check

## How to choose a converter path

1. **Starting from a Spark DataFrame?**
   - Use `make_spark_converter(df, ...)`.
2. **Need TensorFlow input?**
   - Call `converter.make_tf_dataset(...)`.
3. **Need PyTorch input?**
   - Call `converter.make_torch_dataloader(...)`.
4. **Need to clean up cache files?**
   - Call `converter.delete()` or register a custom delete handler if the filesystem needs special treatment.

## Boundary reminders

- `SparkDatasetConverter.PARENT_CACHE_DIR_URL_CONF` must be set before creating a converter.
- The converter caches a materialized parquet version of the DataFrame.
- The cache is reusable across TF or Torch loaders once the converter is created.
- `make_tf_dataset` and `make_torch_dataloader` are context managers that close their readers when the context exits.
- The converter can be used from remote Spark workers only when its cached files are available there too.

## Common tasks

### Create a converter

1. Set `petastorm.spark.converter.parentCacheDirUrl` on the Spark session.
2. Call `make_spark_converter(df)`.
3. Reuse the converter for one or more loaders.

### Build a TensorFlow dataset

1. Call `converter.make_tf_dataset(...)`.
2. Choose `batch_size`, `num_epochs`, and `shuffling_queue_capacity` as needed.
3. Train or evaluate inside the returned context manager.

### Build a PyTorch dataloader

1. Call `converter.make_torch_dataloader(...)`.
2. Choose `batch_size`, `num_epochs`, and `shuffling_queue_capacity` as needed.
3. Use the resulting loader in your model code.

### Delete the cache

1. Call `converter.delete()` after the loader is no longer needed.
2. If the filesystem needs a custom delete implementation, register it first.

## When to stop and switch routes

- If the starting point is raw rows or a schema definition, switch to `create-datasets`.
- If the starting point is an existing dataset instead of a Spark DataFrame, switch to `read-datasets`.