Datasets And Preprocessing

---
name: datasets-and-preprocessing
description: "Guides Open3D-ML dataset loading, split handling, custom dataset
  layouts, and preprocessing validation."
disable-model-invocation: true
metadata:
  disco-role: operating
license: NOASSERTION
---

# Datasets and Preprocessing

Use this sub-skill when you need to understand how Open3D-ML loads point-cloud
datasets, how its dataset classes organize splits, or how to validate a custom
3D dataset layout before training.

## What this sub-skill covers

- Dataset constructors and split APIs.
- Common point-cloud dictionary keys for segmentation and object detection.
- `Custom3D`-style dataset layout and `.npy` file expectations.
- Custom dataset adaptation patterns.
- Preprocessing and bbox-database planning without running expensive source
  scripts or downloads by default.
- Layout validation for small fixtures and malformed examples.

## When to route here

- "How should my dataset folder be structured?"
- "Why is a split empty?"
- "How do I adapt `Custom3D` for my own point cloud data?"
- "What does `get_data()` return for segmentation or object detection?"
- "How do I check whether my `.npy` files are valid before training?"

## Use the bundled helper

Run `scripts/check_dataset_layout.py` to validate a tiny custom dataset root
for split presence, `.npy` shape, and label-column expectations.

## Reading order

1. Read `references/data-formats.md` for the expected dataset and split
   contracts.
2. Read `references/workflows.md` for the common dataset loading and custom
   dataset patterns.
3. Read `references/troubleshooting.md` when a split is empty, a file is
   malformed, or a dataset SDK/download dependency is missing.

## Boundary notes

Include:
- Dataset class selection and split handling.
- Input dictionary shapes and custom dataset layouts.
- Layout and split validation.

Exclude:
- Model training/inference mechanics; use `training-and-pipelines`.
- Visualization and TensorBoard output; use `visualization-and-extensions`.
- Install/backend problems; use `install-and-inspect`.

## Minimal workflow

1. Identify the dataset family and required split names.
2. Confirm the point-cloud dictionary keys and `.npy` layout.
3. Validate a tiny sample directory with `scripts/check_dataset_layout.py`.
4. Hand the dataset outputs to `training-and-pipelines` or
   `visualization-and-extensions`.

## Good handoff signals

A future agent should be able to answer these from this sub-skill alone:

- Which split names are accepted by the dataset base class and custom dataset.
- Which columns are expected for train/val versus test custom point clouds.
- How to recognize a dataset path or label-format error before training.
- Which upstream dataset SDKs are optional versus required for a workflow.