Spatial Raw Processing

---
# AUTO-GENERATED header from skill.yaml — do not edit by hand.
# Edit skill.yaml, then run: python scripts/generate_skill_md.py <skill_dir>
name: spatial-raw-processing
description: Load when converting spatial transcriptomics raw FASTQ pairs through ST-Pipeline into a `raw_counts.h5ad`
  ready for spatial-preprocess. Skip when input is already a count-matrix AnnData (use spatial-preprocess);
  non-spatial bulk / scRNA FASTQ (use bulkrna-read-qc).
version: 0.3.0
author: OmicsClaw
license: MIT
emoji: 🧬
tags:
- spatial
- raw-processing
- fastq
- st-pipeline
- visium
- slideseq
requires:
- anndata
- matplotlib
- numpy
- pandas
- PyYAML
- scipy
- seaborn
---

# spatial-raw-processing

## When to use

The user has paired-end spatial-transcriptomics FASTQ files (`read1` =
spatial barcode + UMI, `read2` = cDNA) plus a STAR genome index, and
wants the standard ST-Pipeline run that produces a `raw_counts.h5ad`
with one row per spatial spot. Single backend: `st_pipeline` (calls
`run_stpipeline` from `skills/spatial/_lib/stpipeline_adapter.py`).

After this skill, chain to `spatial-preprocess` for QC + normalisation.
For non-spatial scRNA FASTQ use `sc-fastq-qc`. For bulk RNA-seq read
QC use `bulkrna-read-qc`.

## Inputs & Outputs

<!-- AUTO-GENERATED from skill.yaml (interface) — do not edit by hand. Regenerate: python scripts/generate_skill_md.py <skill_dir> -->

**Inputs**

- Input kinds: `file`, `directory`
- Modalities: visium, slideseq
- File types: `.fastq`, `.fq`, `.json`, `.yaml`, `.yml`
- FASTQ structure: valid first record; `paired` layout
- Directory layouts (any): `paired-fastq`

**Outputs**

- `tables/gene_qc.csv`
- `tables/raw_gene_qc.csv`
- `tables/raw_processing_run_summary.csv`
- `tables/raw_processing_spatial_points.csv`
- `tables/raw_spot_qc.csv`
- `tables/raw_top_genes.csv`
- `tables/run_summary.csv`
- `tables/saturation_curve.csv`
- `tables/spatial_coordinates.csv`
- `tables/spot_qc.csv`
- `tables/stage_summary.csv`
- `tables/top_genes.csv`
- `figures/raw_detected_genes_spatial.png`
- `figures/raw_spot_qc_histograms.png`
- `figures/raw_top_genes_barplot.png`
- `figures/raw_total_counts_spatial.png`
- `figures/st_pipeline_saturation_curve.png`
- `figures/st_pipeline_stage_attrition.png`
- `omicsclaw_stpipeline_run.json`
- `raw_counts.h5ad`
- `st_pipeline.stderr.txt`
- `st_pipeline.stdout.txt`
- `report.md`
- `result.json`
- Processed AnnData (`saves_h5ad`) — adds `obs`: `barcode`, `x_array`, `y_array`; `obsm`: `spatial`

## Flow

1. Parse args (or load bundle JSON / YAML from positional `--input`).
2. `_apply_effective_defaults` fills missing parameter values (threads, trimming, UMI ranges, etc.).
3. `_validate_real_run_bundle`: check `read1` / `read2` / `ids` / `ref-map` exist and are well-typed; reject duplicate read1=read2; verify FASTQ extension.
4. Call `run_stpipeline(...)` which shells out to ST-Pipeline (requires the `stpipeline` binary on PATH or `--stpipeline-repo` + `--bin-path`).
5. Wrap the resulting count matrix into AnnData with `X = raw_counts`, `layers["counts"]`, `raw = raw_counts_snapshot`.
6. Save `raw_counts.h5ad` and `result.json`. Print "next: spatial-preprocess on raw_counts.h5ad".

## Gotchas

- **All input failures raise typed exceptions wrapped in `SystemExit(1)`.** `spatial_raw_processing.py:353` catches `DataError` / `DependencyError` / `ParameterError` / `ProcessingError` and re-raises as `SystemExit(1)`. The originating raises live in `_validate_real_run_bundle` — `:125` raises `ParameterError(f"Missing required parameter: {key}")` for missing `read1`/`read2`/`ids`; `:128` raises `DataError(...)` for non-existent files; `:131` raises `DataError("Resolved read1/read2 inputs must be FASTQ files.")` for non-FASTQ extensions; `:134` raises `ParameterError` for read1==read2; `:138-141` raises `DataError` for missing / wrong-type STAR index dir; `:145-146` raises `DataError` only when `--ref-annotation` was *provided* but the path is missing or not a file (the param itself is optional — omitting it doesn't raise).
- **`--read1` / `--read2` / `--ids` / `--ref-map` are all required for real runs** (not enforced by argparse `required=True`, validated later). Missing any → `ParameterError`. Demo mode skips this validation entirely.
- **The output filename is always `raw_counts.h5ad`** (`spatial_raw_processing.py:286`). It's not configurable — the contract is consumed by `spatial-preprocess`. Multiple runs to the same `--output` will overwrite.
- **No tables / figures are written.** This skill is a wrapper around an external pipeline; it produces only the AnnData + the upstream tool's logs. `result.json` records the run params, not analysis stats.
- **Demo mode skips ST-Pipeline entirely.** `spatial_raw_processing.py:235` calls `create_demo_upstream_outputs(...)` to fabricate a synthetic `raw_counts.h5ad`. Useful for plumbing checks; does NOT exercise the FASTQ → matrix code path.
- **`--platform` is a metadata label only.** `:201` documents it as "Label recorded in outputs"; ST-Pipeline doesn't branch on it. Common values: `visium`, `visium_hd`, `slideseq`, custom strings.

## Key CLI

```bash
# Demo (synthetic raw_counts.h5ad — does NOT run ST-Pipeline)
python omicsclaw.py run spatial-raw-processing --demo --output /tmp/spatial_raw_demo

# Real run with explicit args
python omicsclaw.py run spatial-raw-processing \
  --read1 sample_R1.fastq.gz --read2 sample_R2.fastq.gz \
  --ids barcodes.tsv \
  --ref-map /refs/star_index_human \
  --ref-annotation /refs/genes.gtf \
  --exp-name visium_001 --platform visium \
  --threads 16 \
  --output results/

# Real run from bundle JSON
python omicsclaw.py run spatial-raw-processing \
  --input run_bundle.json --output results/

# Slide-seq with custom UMI range
python omicsclaw.py run spatial-raw-processing \
  --read1 R1.fq.gz --read2 R2.fq.gz --ids barcodes.tsv \
  --ref-map /refs/star_index --platform slideseq \
  --umi-start-position 1 --umi-end-position 8 \
  --output results/
```

## See also

- `references/parameters.md` — every CLI flag, ST-Pipeline option mapping
- `references/methodology.md` — when ST-Pipeline wins vs Space Ranger; barcode-ID format
- `references/output_contract.md` — `raw_counts.h5ad` schema
- Adjacent skills: `spatial-preprocess` (downstream — required next step; consumes `raw_counts.h5ad`), `bulkrna-read-qc` / `sc-fastq-qc` (parallel — non-spatial FASTQ paths)