Data Ops

---
name: data-ops
description: "Guide dataset download, extraction, and packaging workflows for
  prediction-market-analysis."
metadata:
  disco-role: operating
disable-model-invocation: true
license: MIT
---

# data-ops

Use this sub-skill for dataset download, extraction, archive creation, and host-tool troubleshooting.
It covers the repo's dataset lifecycle but not the analysis or indexing logic.

## Use this route when

- The user wants to download or restore the prebuilt dataset.
- The user wants to package `data/` into `data.tar.zst`.
- The user needs to diagnose `zstd`, downloader, or archive problems.
- The user wants to understand the repo's `make setup` or `make package` behavior.

## Scope

### Included

- Dataset download and extraction from the published archive.
- Packaging `data/` into `data.tar.zst`.
- Host-tool and network troubleshooting for those workflows.
- Archive and sentinel-file behavior.

### Excluded

- Market/trade/block backfills, which belong to `sub-skills/indexing/`.
- Analysis and plotting workflows, which belong to `sub-skills/analysis/`.

## Read first

- `../../references/data-ops.md` for the workflow summary.
- `../../references/data-layout.md` for the directories the dataset should contain.
- `../../references/troubleshooting.md` for cross-cutting failures.

## Core workflow

1. Determine whether the user wants to download/extract or package.
2. Check the required host tools and disk space.
3. Run the download/extract or packaging command from the repo checkout.
4. Verify the resulting `data/` tree or `data.tar.zst` archive.
5. If the workflow fails, use the host-tool or archive troubleshooting notes before retrying.

## Download and extract

The documented path uses `make setup`, which in turn runs the tool-install and download helpers.
Important details:

- `scripts/install-tools.sh` is interactive and may mutate the host.
- `scripts/download.sh` depends on a downloader and `zstd`.
- The archive is large, so timeouts and partial downloads are common failure points.
- A `.download_complete` sentinel marks a successful extraction cycle.

## Packaging

The packaging path is `make package` or `uv run main.py package`.
Important details:

- The archive name is `data.tar.zst`.
- The helper uses `tar --zstd`.
- The code does not remove `data/` after packaging, even though the README wording suggests that behavior.

## Adding or fixing data-ops guidance

- Keep host mutation and network side effects explicit.
- Prefer safe read-only checks before suggesting a download or package rerun.
- Make the archive and sentinel-file semantics obvious.
- Document the exact failure symptoms that mean the host is missing tools rather than the repo being broken.

## Common failure patterns

- Missing `zstd` or downloader tools.
- Not enough disk for the archive or extracted dataset.
- Download links that are temporarily unavailable.
- Confusion about whether packaging should delete the source tree.

## Helpful helper

- `../../scripts/package_data.sh` runs the packaging command from the repo root when the repo environment is available.