geo-data-graph

---
name: geo-data-graph
description: >
  Extract and visualize the data graph for a GEO accession (GSE/GSM/GPL/GDS).
  Resolves the full hierarchy of study, samples, files, and external accessions
  (BioSample, SRA). Optionally downloads supplementary files and produces a
  metadata.csv. Use when the user provides a GEO accession and wants to
  understand or download the dataset structure.
---

# GEO Data Graph

Extract the full data hierarchy of a GEO accession into a structured JSON graph, metadata table, and human-readable tree summary.

## When to Use

- User provides a GEO accession (GSE, GSM, GPL, or GDS)
- User asks to explore, download, or understand a GEO dataset
- User wants to see what samples/files exist in a GEO study before downloading
- User wants to build a metadata.csv from a GEO study

## Quick Start

Activate the conda environment first (see `README.md` for setup):
```bash
conda activate geo-data-graph
```

Script location: `scripts/` (relative to this skill directory)

### Metadata-only mode (default, fast)

Resolve and display the data graph without downloading files:
```bash
python scripts/03_build_data_graph.py \
  --accession GSE204716 \
  --output /path/to/output
```

This produces:
- `{output}/GSE204716/data_graph.json` -- full hierarchical graph
- `{output}/GSE204716/metadata.csv` -- one row per sample
- Terminal tree print (first 5 samples + group summary)

### Download mode

Also download supplementary files:
```bash
python scripts/03_build_data_graph.py \
  --accession GSE204716 \
  --output /path/to/output \
  --download
```

### Filter samples

Download only a subset of samples matching a characteristic:
```bash
python scripts/03_build_data_graph.py \
  --accession GSE204716 \
  --output /path/to/output \
  --download \
  --sample-filter "tissue:kidney"
```

### Dry run

See what would be downloaded without actually downloading:
```bash
python scripts/03_build_data_graph.py \
  --accession GSE204716 \
  --output /path/to/output \
  --download --dry-run
```

## Workflow Rules

### 1. Always resolve first, download second

Run metadata-only mode first to show the user the tree print. Let the user review the data structure and sample groups before deciding to download. Only add `--download` after the user confirms.

### 2. Large study warnings

- **>50 samples**: Print a warning with sample count. Ask user to confirm before downloading.
- **>200 samples**: Suggest using `--sample-filter` to select a subset. Show the sample groups so the user can pick which group to download.

### 3. Show sample groups prominently

The auto-generated `sample_groups` summarize the study design. Present these to the user before the individual sample listing. This helps the user understand the experimental design at a glance.

### 4. Interpreting the output

For accession type details, see `references/accession-types.md`.

**Key points to explain to the user:**
- A GSE study typically contains many GSM samples (not just one)
- Each GSM sample may link to a BioSample (SAMN) and SRA experiment (SRX)
- Supplementary files are the actual data files (count matrices, etc.)
- The `sample_groups` field clusters samples by their most informative characteristics

### 5. Chaining with downstream analysis

The `metadata.csv` can be used directly as input to downstream pipelines (e.g., scAtlasTb, Scanpy, Seurat):
- The `sample` column matches the folder names in the download directory
- All characteristic columns (tissue, disease, etc.) are ready for use as batch/condition variables
- The `biosample` and `sra` columns provide traceability to external databases

### 6. Individual scripts

Each script can be run standalone:

**Resolve only** (script 1):
```bash
python scripts/01_resolve_geo_accession.py \
  --accession GSE204716 --output-json resolved.json
```

**Download only** (script 2, requires resolved JSON):
```bash
python scripts/02_download_and_organize.py \
  --resolved resolved.json --output /path/to/output
```

## Output Files

For the JSON schema, see `references/graph-schema.md`.

### data_graph.json

Hierarchical structure:
```
GSE accession
  +-- Study metadata (title, organism, summary, platform)
  +-- sample_groups (auto-clustered by characteristics)
  +-- samples[] (one per GSM)
  |     +-- characteristics (tissue, disease, donor, etc.)
  |     +-- biosample, sra (external accession links)
  |     +-- files[] (name, size, url, local_path)
  |     +-- download_status
  +-- summary_stats (total files, size, averages)
```

### metadata.csv

One row per GSM sample with columns:
`sample, description, title, source_name, biosample, sra, n_files, total_size_mb, download_status, [all characteristic keys...]`

## Dependencies

- Python 3.10+
- `GEOparse` (for SOFT file parsing)
- `requests` (for HTTP downloads)
- `beautifulsoup4` (for HTML relation scraping)
- `pandas` (optional, for metadata inspection)

## Limitations

- NCBI rate-limits: ~3 requests/second without API key. HTML scraping adds 0.5s delay per sample.
- Very large studies (500+ samples) may take 5-10 minutes to resolve.
- Supplementary file download speed depends on NCBI server load and file sizes.
- GPL and GDS accessions do not have sample-level data; only metadata is returned.