Data Structures And Io

---
name: data-structures-and-io
description: "Design, validate, and troubleshoot MMEngine datasets, transforms,
  samplers, collate functions, data elements, and backend-agnostic file IO."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# MMEngine Data Structures and IO

Use this sub-skill when work mentions `BaseDataset`, `load_data_list`, `full_init`, `Compose`, data pipelines, `DefaultSampler`, `pseudo_collate`, `default_collate`, `BaseDataElement`, `InstanceData`, `PixelData`, `LabelData`, `mmengine.fileio`, `backend_args`, or legacy `FileClient` behavior.

## Read Order

1. `references/dataset-and-transform-workflows.md` — Build and validate dataset annotations, `BaseDataset` subclasses, transform pipelines, samplers, and collate choices.
2. `references/data-elements-and-fileio.md` — Use data element containers and backend-agnostic file IO helpers safely.
3. `references/troubleshooting.md` — Diagnose common dataset, transform, collate, data element, and file backend failures.
4. `scripts/data_contract_smoke.py` — Run a tiny local smoke check for a `BaseDataset` subclass, `InstanceData` constraints, and JSON/text/bytes file IO.

## Scope Boundaries

- Owns `mmengine.dataset`, `mmengine.structures`, `mmengine.fileio`, dataset metadata, lazy/full initialization, serialization, samplers/collation, and backend-agnostic path helpers.
- Route runner config placement for `train_dataloader`, `val_dataloader`, and `test_dataloader` to `../runner-and-training/SKILL.md`.
- Route `BaseModel`, data preprocessor, evaluator, metric, and inference expectations for samples to `../models-metrics-and-inference/SKILL.md`.
- Route registry and config syntax fundamentals for registering datasets/transforms/functions to `../configuration-and-registry/SKILL.md`.

## Fast Workflow

1. Define the sample contract first: raw annotation fields, `metainfo`, transformed data dict keys, data sample objects, and collated batch shape/type.
2. Validate the dataset independently with `len(dataset)`, `dataset.metainfo`, `dataset.get_data_info(0)`, and `dataset[0]` before wiring it into a runner.
3. Choose `pseudo_collate` when downstream code expects lists of per-sample tensors/data elements; choose `default_collate` only when stackable tensor/number/array shapes are consistent.
4. Use `backend_args` and unified `mmengine.fileio` helpers for IO; treat `FileClient` and `file_client_args` as legacy compatibility only.
5. Use the bundled smoke script as a package/API sanity check before debugging larger project-specific data code.