Create Datasets

---
name: "create-datasets"
description: "Routes Petastorm dataset creation, schema design, metadata repair,
  copy-and-filter workflows, and row-group indexing from Spark and existing
  Parquet data."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Create Datasets

Use this route when the task starts from raw rows, a Spark DataFrame, or an existing Parquet store that needs Petastorm metadata,
copying, filtering, schema repair, or row-group indexing.

## Typical triggers

- "How do I create a Petastorm dataset?"
- "How do I define a schema or codec?"
- "How do I copy and filter an existing dataset?"
- "How do I regenerate Petastorm metadata?"
- "How do I build row-group indexes?"
- "How do I use the copy or metadata console tools?"

## What belongs here

- `Unischema` and `UnischemaField`
- `ScalarCodec`, `NdarrayCodec`, `CompressedNdarrayCodec`, `CompressedImageCodec`
- `dict_to_spark_row`
- `materialize_dataset`
- `get_schema` and `get_schema_from_dataset_url`
- `copy_dataset`
- `generate_petastorm_metadata`
- `build_rowgroup_index`
- `SingleFieldIndexer` and `FieldNotNullIndexer`
- filesystem URL resolution used while writing or repairing datasets
- the `petastorm-copy-dataset.py` and `petastorm-generate-metadata.py` commands

## What does not belong here

- consuming an already-created dataset
- TensorFlow and PyTorch adapter details beyond writer-side shape or schema concerns
- Spark DataFrame caching into reusable loaders

Use `sub-skills/read-datasets/` for reader work.
Use `sub-skills/spark-converter/` when the task is caching a Spark DataFrame for later loading.

## Read this first

- `references/workflows.md` for writing, copying, metadata repair, and indexing recipes
- `references/api-reference.md` for verified signatures and behavior
- `references/cli-reference.md` for the dataset copy and metadata commands
- `references/data-formats.md` for the on-disk layout and metadata keys
- `references/troubleshooting.md` for schema, Spark, URL, and metadata failures
- `scripts/smoke_make_minimal_dataset.py` for a tiny write smoke test
- `scripts/smoke_copy_dataset.py` for a copy-and-filter smoke test
- `scripts/smoke_generate_metadata.py` for a metadata-repair smoke test

## How to choose a creation path

1. **Starting from rows or a DataFrame?**
   - Define a `Unischema` and write through `materialize_dataset`.
2. **Need to rewrite an existing dataset?**
   - Use `copy_dataset` for subset/filter/repartition jobs.
3. **Need to restore reader compatibility?**
   - Use `generate_petastorm_metadata`.
4. **Need row-group acceleration?**
   - Use `build_rowgroup_index` with the matching indexer type.

## Boundary reminders

- Non-scalar fields need a codec.
- `materialize_dataset` is the writer-side contract that stamps Petastorm metadata onto the Parquet output.
- `copy_dataset` can narrow columns and drop rows with nulls in selected fields.
- `generate_petastorm_metadata` can infer the schema from the dataset when metadata already exists, or use a supplied unischema class string.
- `build_rowgroup_index` expects compatible indexers and a writable dataset metadata location.

## Common tasks

### Create a new dataset

1. Define a schema with `Unischema`.
2. Convert dictionaries into Spark rows with `dict_to_spark_row`.
3. Wrap the write in `materialize_dataset`.
4. Write to a `file://`, `hdfs://`, `s3://`, or compatible URL.

### Repair or regenerate metadata

1. Open the existing dataset URL.
2. Call `generate_petastorm_metadata` or the CLI wrapper.
3. Reopen the dataset with `make_reader` to confirm the repair.

### Copy or filter a dataset

1. Decide which columns should survive the copy.
2. Decide which fields must remain non-null.
3. Optionally repartition before write.
4. Read the result back to confirm the shape.

### Build a row-group index

1. Create an indexer object for the field you want to accelerate.
2. Call `build_rowgroup_index` from a Spark context.
3. Use the resulting selector on the read side.

## When to stop and switch routes

- If the user only wants to open the data, switch to `read-datasets`.
- If the user wants to convert a Spark DataFrame into a reusable cached loader, switch to `spark-converter`.