1553 Custom Datasets F87d71f2

# Adding Custom Datasets

M4 supports any PhysioNet dataset. This guide shows how to add your own.

## Quick Start: JSON Definition

Create a JSON file in `m4_data/datasets/`:

**Example: `m4_data/datasets/mimic-iv-ed.json`**
```json
{
  "name": "mimic-iv-ed",
  "description": "MIMIC-IV Emergency Department Module",
  "file_listing_url": "https://physionet.org/files/mimic-iv-ed/2.2/",
  "subdirectories_to_scan": ["ed"],
  "primary_verification_table": "mimiciv_ed.edstays",
  "requires_authentication": true,
  "bigquery_project_id": "physionet-data",
  "bigquery_dataset_ids": ["mimiciv_ed"],
  "modalities": ["TABULAR"],
  "schema_mapping": {"ed": "mimiciv_ed"},
  "bigquery_schema_mapping": {"mimiciv_ed": "mimiciv_ed"}
}
```

Then initialize:
```bash
m4 init mimic-iv-ed --src /path/to/your/csv/files
```

## JSON Fields Reference

| Field | Required | Description |
|-------|----------|-------------|
| `name` | Yes | Unique identifier (used in `m4 use <name>`) |
| `description` | Yes | Human-readable description |
| `file_listing_url` | No | PhysioNet URL for auto-download (demo datasets only) |
| `subdirectories_to_scan` | No | Subdirs containing CSV files (e.g., `["hosp", "icu"]`) |
| `primary_verification_table` | Yes | Table to verify initialization succeeded |
| `requires_authentication` | No | `true` if PhysioNet credentialing required |
| `bigquery_project_id` | No | GCP project for BigQuery access |
| `bigquery_dataset_ids` | No | BigQuery dataset IDs |
| `modalities` | No | Data types in this dataset (see below). Defaults to `["TABULAR"]` |
| `schema_mapping` | No | Maps filesystem subdirectories to canonical schema names (see below) |
| `bigquery_schema_mapping` | No | Maps canonical schema names to BigQuery dataset IDs (see below) |

### Available Modalities

| Modality | Description | Available Tools |
|----------|-------------|-----------------|
| `TABULAR` | Structured tables (labs, demographics, vitals, etc.) | `get_database_schema`, `get_table_info`, `execute_query` |
| `NOTES` | Clinical notes and discharge summaries | `search_notes`, `get_note`, `list_patient_notes` |

Tools are filtered based on the dataset's declared modalities. If not specified, defaults to `["TABULAR"]`.

### Schema Mapping (Canonical Table Names)

M4 uses canonical `schema.table` names (e.g., `mimiciv_hosp.patients`) that work identically on both DuckDB and BigQuery backends. The `schema_mapping` and `bigquery_schema_mapping` fields control how these canonical names are constructed.

**`schema_mapping`** maps filesystem subdirectories to canonical schema names. When DuckDB creates views, files from each subdirectory are placed into the corresponding schema:

```json
{
  "schema_mapping": {
    "hosp": "mimiciv_hosp",
    "icu": "mimiciv_icu"
  }
}
```

With this mapping, a file at `hosp/patients.csv` becomes queryable as `mimiciv_hosp.patients`.

For datasets where all files are in the root directory (no subdirectories), use an empty string key:

```json
{
  "schema_mapping": {
    "": "eicu_crd"
  }
}
```

**`bigquery_schema_mapping`** maps canonical schema names to BigQuery dataset IDs. This allows the BigQuery backend to translate canonical names to the actual GCP dataset names:

```json
{
  "bigquery_schema_mapping": {
    "mimiciv_hosp": "mimiciv_hosp",
    "mimiciv_icu": "mimiciv_icu"
  }
}
```

With this, a query for `mimiciv_hosp.patients` is rewritten to `physionet-data.mimiciv_hosp.patients` on BigQuery.

Custom datasets without `schema_mapping` still work — tables will be created with flat names in the `main` schema (backward-compatible behavior).

## Initialization Process

When you run `m4 init <dataset>`:

1. **Download** (if `file_listing_url` exists and files missing)
2. **Convert** CSV.gz files to Parquet format
3. **Create** DuckDB views over the Parquet files
4. **Verify** by querying `primary_verification_table`

## Directory Structure

M4 organizes data like this:

```
m4_data/
├── datasets/           # Custom JSON definitions
│   └── my-dataset.json
├── raw_files/          # Downloaded CSV.gz files
│   └── my-dataset/
│       └── *.csv.gz
├── parquet/            # Converted Parquet files
│   └── my-dataset/
│       └── *.parquet
└── databases/          # DuckDB databases
    └── my_dataset.duckdb
```

## Using Existing CSV Files

If you already have CSV files (either `.csv` or `.csv.gz`), point to them with `--src`:

```bash
m4 init my-dataset --src /path/to/csvs
```

M4 will:
1. Convert CSV/CSV.gz files to Parquet format
2. Create DuckDB views
3. Set the dataset as active

## Credentialed Datasets

For datasets requiring PhysioNet credentials (most full datasets):

1. Get credentialed access on PhysioNet
2. Download manually using wget:
   ```bash
   wget -r -N -c -np --user YOUR_USERNAME --ask-password \
     https://physionet.org/files/dataset-name/version/ \
     -P m4_data/raw_files/dataset-name
   ```
3. Initialize:
   ```bash
   m4 init dataset-name
   ```

## Programmatic Registration

For more control, register datasets in Python:

```python
from m4.core.datasets import DatasetDefinition, DatasetRegistry, Modality

my_dataset = DatasetDefinition(
    name="my-custom-dataset",
    description="My custom clinical dataset",
    primary_verification_table="patients",
    modalities=frozenset({Modality.TABULAR}),
)

DatasetRegistry.register(my_dataset)
```

## Tips

- **Start with demo data:** Test your setup with `mimic-iv-demo` first
- **Check table names:** Use `get_database_schema` tool to see available tables
- **Verify initialization:** `m4 status` shows if Parquet and DuckDB are ready
- **Force reinitialize:** `m4 init <dataset> --force` recreates the database