Yume Extract Author

---
name: yume-extract-author
description: >-
  Stage 5 of the text-to-world pipeline: write a per-scene Python script (on
  lib_extract.py) that turns a semantic map + class catalog + heightmap into
  extracted.json. Use inside /yume-create-scene after the semantic map
  exists.
---

# /yume-extract-author

You are the **stage-5 author** for Yume's text-to-world pipeline.
You read a SEMANTIC MAP + CLASS CATALOG (and optionally a HEIGHTMAP),
then you write a **custom Python script** that extracts entity
positions / biome masks / rotations from the semantic map and emits
the final `extracted.json` that drives the rest of the pipeline.

> **⚠ CURRENT PIPELINE (2026-05-27).** The per-scene-LLM-script flow
> below is the ORIGINAL extraction approach. It has been largely
> SUPERSEDED by a deterministic strategy-library path:
> `tools/visual_layout/compose_world.py` reads
> `data/lib/extraction_strategies.json` (per-class strategy: extraction
> method, rotation rule, fit-to-mask, variant buckets, kit mesh, albedo
> override) and dispatches via `lib_extract_dispatch.dispatch_extraction` —
> no per-scene script needed. compose_world writes the MAP layer
> (entity defs + placements + biome ground + water + roads); a separate
> `compose_shell.py` adds the presentation layer (camera/player/input/
> lighting/.tscn) so a map generator never decides how you view or
> control the world. Buildings render as **kit-of-parts composite
> meshes** (floor-tiered houses, townhall/bridge/wall kits) — see
> `yume-asset-designer` `reference/asset-strategies.md` § Strategy C2. New classes = new JSON entries
> in `extraction_strategies.json` (no code). Backups NEVER go in the
> engine-globbed `entities/` dir — use `<game>/_snapshots/` (see
> `.claude/rules/data-demo.md`). This skill remains the reference for
> the LLM-as-parser fallback when a scene's extraction needs custom
> routing the strategy library can't express.

## When to invoke

- After stage 2 (catalog) and stage 3 (semantic map) and stage 4
  (heightmap) have all produced their artifacts
- Before stage 6 (asset prompts) or stage 7 (engine wiring)

## When NOT to invoke

- For HUD / screen extraction — those use compose_hud / compose_screen
- For raw photoreal-to-flat conversion — that's the SAME stage 5
  pipeline but the prompt+harness is what processes it; this skill
  writes the script, not the prompt

## Inputs

1. **Semantic map PNG** — stage 3 output (flat-color, machine-readable)
2. **Class catalog JSON** — stage 2 output (`/tmp/_class_catalog.json`)
3. **Heightmap PNG** (optional) — stage 4 output, for entities that
   need to read terrain elevation
4. **World size in meters** — typically 80×80m for a Yume town scene
   (consults scene.json or defaults)

## Output

A single file: `/tmp/_extracted.json` with the structured tile-map.

Schema:

```json
{
  "source_semantic_map": "path",
  "source_catalog": "path",
  "source_heightmap": "path or null",
  "image_size": [W, H],
  "world_size_meters": [80.0, 80.0],
  "coverage_pct": 99.7,
  "unassigned_pct": 0.3,
  "classes": [
    {
      "name": "grass",
      "intent_type": "terrain_shader",
      "hex": "#a0d870",
      "biome_mask_path": "/tmp/biome_grass.png",
      "coverage_pct": 32.1,
      "expected_coverage_pct": 30.0,
      "shader_slot_hint": 0
    },
    {
      "name": "house",
      "intent_type": "object_placement",
      "hex": "#a04020",
      "instance_count": 47,
      "expected_count": 60,
      "instances": [
        {
          "id": "house_001",
          "position_px": [320, 410],
          "position_world": [-5.63, 6.09],
          "bbox_px": [310, 400, 330, 420],
          "size_px": [20, 20],
          "size_world": [1.56, 1.56],
          "rotation_deg": 12.0,
          "rotation_confidence": 0.71,
          "area_px": 380
        }
      ]
    },
    {
      "name": "valley",
      "intent_type": "terrain_displacement",
      "heightmap_ref": "/tmp/heightmap.png",
      "displacement_meters": [0, 5]
    }
  ]
}
```

## The procedure

### Step 1 — Read inputs

```bash
cat /tmp/_class_catalog.json  # peek at classes + hexes
file /tmp/semantic_map.png   # confirm exists
```

### Step 2 — Write the per-scene extraction script

The script lives at `/tmp/_extract_<short_hash>.py` (ephemeral).
It uses `tools.visual_layout.lib_extract` for all heavy lifting.

Skeleton:

```python
#!/usr/bin/env python3
\"\"\"Per-scene extraction script — generated by yume-extract-author.
Catalog: <catalog hash>
Semantic map: <semantic_map hash>
\"\"\"
import json, sys
from pathlib import Path
sys.path.insert(0, "<repo_root>")
from tools.visual_layout import lib_extract as L

CATALOG_PATH    = Path("/tmp/_class_catalog.json")
SEMANTIC_PATH   = Path("/tmp/semantic_map.png")    # or wherever stage 3 wrote it
HEIGHTMAP_PATH  = Path("/tmp/heightmap.png")        # optional
OUT_PATH        = Path("/tmp/_extracted.json")
MASKS_DIR       = Path("/tmp/biome_masks")
WORLD_SIZE_M    = (80.0, 80.0)   # configurable; check scene.json if exists

# ----- Step A: load -----
catalog = json.loads(CATALOG_PATH.read_text())
img = L.load_rgb(SEMANTIC_PATH)
H, W, _ = img.shape

# ----- Step B: hard partition via nearest-palette ----- 
# Only the classes that ARE in the semantic map color channel.
# terrain_displacement classes are NOT in the palette (they live
# in the heightmap), so filter them out here.
color_palette = [
    (cls["name"], cls["hex"])
    for cls in catalog["classes"]
    if cls["intent_type"] != "terrain_displacement"
]
labels = L.threshold_nearest_palette(img, color_palette)

# ----- Step C: per-class extraction -----
classes_out = []
for idx, cls in enumerate([c for c in catalog["classes"]
                            if c["intent_type"] != "terrain_displacement"]):
    name = cls["name"]
    mask = (labels == idx)
    n_px = int(mask.sum())
    pct  = 100 * n_px / (H * W)
    entry = {
        "name": name,
        "hex":  cls["hex"],
        "intent_type": cls["intent_type"],
    }

    if cls["intent_type"] == "terrain_shader":
        # Save the mask + emit biome metadata
        mask_path = MASKS_DIR / f"biome_{name}.png"
        L.save_mask(mask, mask_path)
        entry["biome_mask_path"] = str(mask_path)
        entry["coverage_pct"] = round(pct, 3)
        entry["expected_coverage_pct"] = cls.get("expected_coverage_pct")
        entry["shader_slot_hint"] = idx
        classes_out.append(entry)

    elif cls["intent_type"] == "object_placement":
        # Find connected components, infer rotation per instance
        comps = L.connected_components(mask, connectivity=4, min_area=20)
        instances = []
        for i, comp in enumerate(comps):
            angle_deg, elongation = L.infer_rotation_deg(comp)
            # If elongation < 1.2 (roughly isotropic), rotation is
            # noise → set to 0
            if elongation < 1.2:
                angle_deg = 0.0
            x0, y0, x1, y1 = comp["bbox"]
            cx, cy = comp["centroid"]
            wx, wz = L.pixel_to_world(cx, cy, (W, H), WORLD_SIZE_M)
            size_world_x, _ = L.pixel_to_world(x1 - x0, 0, (W, H), WORLD_SIZE_M)
            _, size_world_z = L.pixel_to_world(0, y1 - y0, (W, H), WORLD_SIZE_M)
            instances.append({
                "id": f"{name}_{i+1:03d}",
                "position_px":    [int(cx), int(cy)],
                "position_world": [wx, wz],
                "bbox_px":        [x0, y0, x1, y1],
                "size_px":        [x1 - x0 + 1, y1 - y0 + 1],
                "size_world":     [round(abs(size_world_x - L.pixel_to_world(0, 0, (W, H), WORLD_SIZE_M)[0]), 3),
                                   round(abs(size_world_z - L.pixel_to_world(0, 0, (W, H), WORLD_SIZE_M)[1]), 3)],
                "rotation_deg":   angle_deg,
                "rotation_confidence": round(min(elongation - 1, 1.0), 2),
                "area_px":        comp["area_px"],
            })
        entry["instances"] = instances
        entry["instance_count"] = len(instances)
        entry["expected_count"] = cls.get("expected_count")
        classes_out.append(entry)

# ----- Step D: heightmap reference for displacement classes -----
for cls in catalog["classes"]:
    if cls["intent_type"] == "terrain_displacement":
        entry = {
            "name": cls["name"],
            "intent_type": "terrain_displacement",
            "heightmap_ref": str(HEIGHTMAP_PATH) if HEIGHTMAP_PATH.exists() else None,
            "displacement_meters": catalog["heightmap_hints"]["elevation_range_estimate_meters"],
        }
        classes_out.append(entry)

# ----- Step E: write extracted.json -----
L.write_extracted_json(
    OUT_PATH,
    semantic_map_path=str(SEMANTIC_PATH),
    catalog_path=str(CATALOG_PATH),
    image_size=(W, H),
    world_size_m=WORLD_SIZE_M,
    classes=classes_out,
    coverage_pct=100.0,           # nearest-palette gives 100% by construction
    unassigned_pct=0.0,
    notes=f"Extracted by per-scene script from yume-extract-author. {len(classes_out)} classes.",
)
print(f"wrote {OUT_PATH}")
```

### Step 3 — Run the script

```bash
python3 /tmp/_extract_<hash>.py
```

### Step 4 — Validate the output

Read `/tmp/_extracted.json`. Check:
- **Coverage** — `coverage_pct` should be 100.0 (nearest-palette
  partition is exhaustive by construction)
- **Class counts** vs `expected_count` from the catalog — drift > 50%
  in either direction is a yellow flag (LLM mis-counted in stage 2,
  or stage 3 generated too few/many)
- **Coverage drift** for terrain_shader classes — drift > 30% from
  `expected_coverage_pct` is a yellow flag
- **Per-instance sanity** — bbox sizes shouldn't all be 1px (= tiny
  noise components), shouldn't all be 90% of the image (= over-merged)

### Step 5 — Iterate if needed

If validation reports issues:
- Wrong color tolerance → adjust `threshold_by_hex` calls
- Components too small → bump `min_area` to drop noise
- Components too merged → drop `min_area` or use connectivity=8

The per-scene script is ephemeral — just rewrite it. Don't try to
make the script handle every edge case generically; the LIBRARY
handles the generic cases, the per-scene script handles the
per-scene routing.

## Custom per-scene logic

Most scenes work with the skeleton above unchanged. But some need
custom handling:

### Multi-color "family" classes

If the catalog has `small_house`, `large_house`, `townhall` all in
the red-brown family, but you want extraction to group them as a
"building" family with size-subclassing, modify Step C:

```python
# Group house family
house_family = ["small_house", "medium_house", "large_house", "townhall"]
house_components = []
for fam_name in house_family:
    fam_idx = palette_names.index(fam_name)
    fam_mask = (labels == fam_idx)
    for comp in L.connected_components(fam_mask):
        comp["family_member"] = fam_name
        house_components.append(comp)
# emit as a single "building" class with subclass tag per instance
```

### Class with required minimum count

If a class MUST have at least N instances (e.g. "tower" must have
≥8 for an octagonal wall), validate during extraction and emit
a warning JSON field:

```python
if cls["name"] == "tower" and len(instances) < 8:
    entry["_warning"] = f"expected ≥8 towers, found {len(instances)} — wall may be incomplete"
```

### Connected-component splitting

Sometimes a class's pixels form ONE big blob that should be split
(e.g. a long market row that the model emitted as one connected
patch). Handle by:
- After `connected_components`, if a component's bbox aspect ratio
  is extreme (>5:1), split it into N equal segments along the long
  axis.

## Strict rules

1. **The library handles primitives; the script handles routing.**
   Don't reimplement color thresholding / flood fill in the
   per-scene script — call `lib_extract.threshold_by_hex` /
   `connected_components` / `infer_rotation_deg`.

2. **Output JSON schema is fixed.** All downstream consumers (stage
   6 asset gen, stage 7 engine wiring) depend on the field names
   listed in the Output section above. Don't rename fields.

3. **rotation_deg = 0 when elongation < 1.2.** The PCA-derived
   angle is meaningless for isotropic shapes (circles, squares).
   Always check elongation and zero-out if low.

4. **Coordinate convention: origin = center of map.** `world_size_m`
   defines the world extent in meters; pixel (W/2, H/2) maps to
   world (0, 0). Confirms with the engine's convention
   (`position_scale=1.0` for aldenmere → world coords in meters).

5. **terrain_displacement has NO color in the semantic map.** It
   lives in the heightmap. Emit a heightmap_ref entry per
   terrain_displacement class without trying to threshold colors.

## Worked example

Run on the medieval-town stage-3 output (semantic_town_via_edits.png)
+ stage-2 catalog (`_class_catalog.json` with 12 classes). The
resulting extracted.json should have:
- 6 terrain_shader entries with biome_mask paths + coverage %
- 6 object_placement entries with per-instance positions:
  - house: ~50-70 instances (catalog expected 60)
  - wall_segment: ~8-12 instances (catalog expected 8)
  - tower: ~10-14 instances (catalog expected 12)
  - bridge: ~2 instances
  - fountain: 1 instance
  - townhall: 1 instance
- Total coverage 100%

The bbox sizes + centroids feed stage 6 (asset prompts) — each
unique class becomes one asset prompt template; the per-instance
data becomes scene placements.

## What this skill is NOT

- NOT a generator of the semantic map (stage 3)
- NOT a generator of the catalog (stage 2)
- NOT a generator of asset prompts (stage 6)
- NOT an engine integrator (stage 7)
- NOT a member of the locked 2D pipelines (HUD/screen)

## Reference files

- `tools/visual_layout/lib_extract.py` — generic helpers (this skill's library)
- `.claude/skills/yume-scene-class-catalog/SKILL.md` — stage 2 producer of the catalog this consumes
- `godot/data/lib/semantic_palette.json` — the conventions backing the catalog