Memory Snapshots

---
name: reference_hf_dataset_publishing_skill
description: "Reusable ecosystem skill for saving/publishing any analysis's results to Hugging Face as a queryable dataset — location, what it does, commit-pending status"
metadata: 
  node_type: memory
  type: reference
  originSessionId: 87996704-81da-4051-a504-a26eab248280
---

**2026-07-11 (built during World Energy HF publish, [[project_world_energy_field_explorer_program]]).**
New ecosystem skill: **`hf-dataset-publishing`** at
`workspace-hub/.claude/skills/data/hf-dataset-publishing/` — 3 files:
- `SKILL.md` — reshape nested analysis results → flat parquet tables → dataset card with viewer
  `configs:` block + sha256 provenance → publish to `aceengineer/<repo>-<projection>` → verify via
  datasets-server `/rows` API.
- `publish_analysis_to_hf.py` — bundled helper (auth check → create_repo/upload_folder → poll
  `/is-valid`; `--dry-run` + PRIVATE-by-default fail-safe; compiles clean).
- `PROMPT.md` — copy-paste parameterized prompt (`<analysis>`/`<repo>`/`<projection>`).

Load-bearing features encoded: the **data-quality gate** ("faithful to source ≠ correct" — sanity-check
VALUES not row-counts; withhold implausible columns + file a `cat:data` issue — the wed#971 catch);
**keep-separate-from-`aceengineer/<repo>-runs`** (that's wh#3433's contract-managed algorithm-run
ledger); **license routing** (public-domain federal → public+cc-by-4.0; vendor/private → NOT public;
per `.claude/rules/codes-standards-data-routing.md`); sandbox gotchas (`python3 -c`/`base64 -d` DENIED
→ heredocs/`gh api` raw; CDN lag → verify via `gh api`/`HfApi`).

**GENERIC SCRIPT (preferred entry point for other sessions):**
`workspace-hub/scripts/hf/save_results_to_hf.py` — one CLI, any repo/algorithm output.
`python save_results_to_hf.py --repo-id aceengineer/<repo>-<projection> --input <json|csv|parquet|dir> [--public] [--source-repo o/r] [--algorithm name@ver] [--dry-run]`.
Auto-discovers tables from arbitrary nested JSON (every list-of-dicts + dict-of-dicts, recursing wrappers; nested scalars→dotted cols; deeper→JSON-stringified lossless), sanitizes NaN/inf→native parquet null, writes viewer card w/ `configs:`+sha256 provenance, publishes **PRIVATE by default**, verifies via datasets-server, prints **per-column numeric stats** (the data-quality eyeball — it auto-surfaced the #971 $380 breakeven in testing). **REFUSES `-runs` targets.** Dry-run tested on the nested explorer bundle → 6 tables auto-found.

**STATUS: MERGED — PR wshub #3473 (2026-07-11 23:48Z), script live on main (`scripts/hf/save_results_to_hf.py`).** Committed via GitHub git-data API (blobs→tree→commit→ref) because local git + the skill-index generator both HANG on the FUSE mount. Follow-up **#3474**: regen `config/agents/skill-index-full.yaml` from a non-FUSE checkout (advisory Skill-Index-Coherence check red until then; not merge-blocking — main unprotected). All sessions get the tool via sync. Proven end-to-end by the live publish of `aceengineer/worldenergydata-explorer`. Linter later added `--card-note` (disclose withheld columns) — keep.