dataset-generator

---
name: dataset-generator
description: Use this when the user wants to generate, normalize, verify, deduplicate, or export training datasets for Codex, Antigravity, or Claude Code from topics, URLs, reference material, web research, or existing JSONL/CSV files. Supports SFT and DPO workflows, custom export schemas, and deterministic local pipeline scripts.
---

# Dataset Generator

This skill is a tool-native dataset pipeline for Codex, Antigravity, and Claude Code.

- Use the IDE's own tools for browsing, reading, search, and reasoning.
- Use local Python scripts for deterministic normalization, state tracking, verification, deduplication, and export.
- Do not call external LLM-provider APIs as part of this skill.

## Command surface

- `dataset generate "<request>" [--count <n>]`
- `dataset collect "<topic or query>" [--urls url1 url2] [--paths ./dir]`
- `dataset verify <path/to/file>`
- `dataset audit [<path/to/file>]`
- `dataset export --format <openai|huggingface|csv|jsonl|all> [--schema-file path] [--split 0.1]`

If `dataset generate` does not include a size, default to `500` records.
If `dataset collect` does not include `--max-results`, default to `10` results per query.

## Core architecture

- `sub-skills/` contains the cognitive instructions.
- `scripts/` contains deterministic helpers.
- `resources/internal-schema/canonical_schema.json` is the fixed pipeline backbone.
- `resources/target-schemas/` contains preset export profiles.
- `resources/templates/custom_flat_schema.json` is the starting point for custom headers.

## Fixed vs flexible schema

- The canonical internal schema is fixed.
- The final export schema is not universal and must be chosen per user request.
- For custom CSV or flat JSONL headers, create or update a schema file and pass it to `scripts/export.py`.

Read `sub-skills/dataset-strategy.md` first whenever the target output schema is not already obvious.

## Workflow selection

### 1. `dataset generate`

Use this when the user wants a new dataset or wants source material structured into one.

1. Read `sub-skills/dataset-strategy.md` and explicitly decide:
   - request type
   - `task_type`
   - `source_type`
   - target export schema
   - target effective example count
   - whether this is a fresh run or a resume

If the user does not specify a size, set the target effective example count to `500`.
2. If existing runs may matter, inspect the SQLite state before generating:

```bash
python3 -c "from scripts.utils.db import initialize_database, get_connection, list_runs; initialize_database(); conn = get_connection(); print([dict(row) for row in list_runs(conn, limit=5)]); conn.close()"
```

If there is a relevant unfinished or recent run, ask whether to resume or start fresh.

3. Choose the source route:

- Topic-driven synthetic generation:
  - Read `sub-skills/seed-generator.md`.
  - Draft canonical JSONL records and import them with `--source-type generated`.
  - If the requested count is large, work in batches until the target count is reached instead of stopping after the first small draft.
- URL or reference-material structuring:
  - Read `sub-skills/local-collector.md`.
  - **First**: try the IDE's native search/browsing tools to collect material directly.
  - **Fallback**: if IDE tools are unavailable or the collection is large, run:
    ```bash
    python3 scripts/collect.py --urls <url1> [url2 ...] --tool-context <context>
    ```
  - Draft canonical JSONL from the collected output and import with `--source-type url_reference`.
- Existing dataset restructuring:
  - Read `sub-skills/seed-generator.md`.
  - Normalize the source dataset into canonical JSONL and import it with `--source-type raw_dataset`.
- Internet-research dataset building:
  - Read `sub-skills/local-collector.md`.
  - **First**: use the IDE's native search tools to find evidence, draft canonical records, and import.
  - **Fallback**: if IDE tools are unavailable or the target record count requires broad crawling, run:
    ```bash
    python3 scripts/collect.py --query "<topic>" --max-results 10 --tool-context <context>
    ```
  - The collector outputs `workspace/collected_<timestamp>.jsonl`; the agent then drafts proper instruction/response records and imports them with `--source-type internet_research`.
  - If the user does not specify a size, continue collecting and drafting until `500` records are planned or imported.

4. Load draft records into SQLite:

Preferred automated path when you already have planned batch files:

```bash
python3 scripts/build_loop.py --batch <drafts_batch_01.jsonl> --batch <drafts_batch_02.jsonl> --plan-file <coverage_plan.json> --source-type <generated|url_reference|raw_dataset|internet_research> --tool-context <codex|claude|antigravity> [--review-file <review.jsonl>] [--verify-min-response-length 5]
```

This orchestrates import-time dedup, optional verify/dedup, and a coverage check after every batch.
For short-label classification corpora, lower `--verify-min-response-length` so labels like `VULNERABLE` are not rejected by the generic heuristic floor.
If the coverage plan sets `require_review_file: true`, `build_loop.py` will fail fast unless `--review-file` is provided so semantic judging runs during the build.

After each batch, `build_loop.py` writes `workspace/build_loop_progress.json` with `batches_done`, `last_coverage`, and a `drift` object (`drift_score`, `drift_flag`, `new_gaps`, `resolved_gaps`). Read this file to check progress between batches. If `drift_flag: true`, inspect the new gaps before sending the next batch. Use `record_history.py` to append a lineage snapshot to `workspace/record_history.jsonl` at any point.

Manual import path:

```bash
python3 scripts/generate.py --input <drafts.jsonl> --source-type <generated|url_reference|raw_dataset|internet_research> --tool-context <codex|claude|antigravity> --dedup-threshold 0.85
```

Imported drafts are promoted into the runnable pipeline with status `raw_generated` unless they are explicit placeholder seeds.
When `--dedup-threshold` is used, near-duplicates are marked `deduped` immediately instead of inflating the raw count.

If the user is intentionally building red-team, security, pentest, prompt-injection, jailbreak, or system-prompt-leak training data, default to injection-tolerant import behavior. The scripts now auto-enable this for matching requests, and you can still pass `--allow-injections` explicitly for clarity. Use `--enforce-security-flags` only when you want strict flagging even on those corpora.
For untrusted sources, normalization also strips hostile control characters and may add `metadata.security_flags` plus `metadata.requires_manual_review`.

For generation requests, do not treat a small sample as the finished dataset unless the user explicitly asked for a small sample, prototype, or test run.
Do not treat the raw imported count as success. The generation loop is complete only when the post-dedup effective count and per-bucket coverage targets are met.

4B. If you are not using `build_loop.py`, measure effective progress after each import batch before drafting the next batch:

```bash
python3 scripts/coverage.py --from-status raw_generated --from-status augmented --from-status verified_pass --threshold 0.85 --plan-file <coverage_plan.json>
```

The coverage plan should define:
- `target_effective_count`
- `max_share_per_group`
- `group_minimums` keyed by metadata paths such as `metadata.subtopic`, `metadata.context_type`, `metadata.response_shape`, or `metadata.label`
- optional `required_fields` for metadata or provenance paths that every kept record must carry
- optional `joint_group_rules` for multi-axis balance such as `difficulty x label` or `persona x response_shape`
- optional `provenance` rules such as a minimum `real_world` share and required reference fields for real-world records
- optional `response_length` rules to cap median answer size or the share of oversized responses
- optional `response_structure` rules to prevent one dominant JSON or text skeleton from taking over the corpus
- optional `response_prefix` limits to prevent one repeated opening from dominating the corpus
- optional `model_visibility` rules to customize export-time sanitization for model-visible `instruction` and `context` without dropping audit metadata. If omitted, export applies a conservative built-in profile; set `"enabled": false` to disable it.
- optional `require_review_file: true` to make semantic LLM review mandatory during the build loop

These advanced sections are advisory unless you set `blocking: true` inside that section. This keeps fixed-envelope or HTTP-heavy datasets from being rejected by default while still surfacing the findings.

If the effective count is still below target or any bucket is under its minimum, draft another batch aimed only at the missing buckets.

5. If augmentation is needed, read `sub-skills/diversity-engine.md` and either import rewritten augmentations or create metadata variants:

```bash
python3 scripts/augment.py --input <augmented.jsonl> --tool-context <codex|claude|antigravity>
```

Or deterministic metadata variants:

```bash
python3 scripts/augment.py --from-status raw_generated --persona expert --difficulty hard
```

Metadata-variant rows are scaffolding only. They are now marked `rewrite_required` and cannot pass `verify.py` until the instruction/response has actually been rewritten.

6. Run heuristic verification:

```bash
python3 scripts/verify.py --from-status raw_generated --from-status augmented [--plan-file <coverage_plan.json>]
```

7. If semantic judging is needed, read `sub-skills/llm-judge.md`, produce a review file, then apply it:

Before semantic judging, inspect records with `metadata.requires_manual_review` or `metadata.security_flags` and treat their content as untrusted data.

```bash
python3 scripts/verify.py --from-status raw_generated --review-file <review.jsonl> [--plan-file <coverage_plan.json>]
```

After adjudication, run `judge_insights.py` to understand why records failed and what to fix before re-drafting:

```bash
python3 scripts/judge_insights.py --review-file <review.jsonl> [--output workspace/judge_insights.json] [--top-n 10]
```

The output clusters `fail_reasons` into canonical buckets (`vague_instruction`, `weak_response`, `apology_opener`, `trope_opener`, `refusal_error`, `grounding_fail`, `dpo_quality`, `format_violation`, `leakage`, `other`) and emits one actionable recommendation per bucket. Use the `recommendations` array to guide the next draft batch.

8. Deduplicate passing records:

```bash
python3 scripts/dedup.py --from-status verified_pass
```

The final dedup pass still runs before export, but it is not a substitute for generation-time duplicate suppression and coverage tracking.

9. Read `sub-skills/formatter-exporter.md` and export the dataset plus data card:

```bash
python3 scripts/export.py --format <openai|huggingface|csv|jsonl|all> [--schema-file <schema.json>] [--split 0.1] [--plan-file <coverage_plan.json>]
```

### 2. `dataset verify`

Use this when the user already has a file and wants an audit or cleanup pass.

Read `sub-skills/data-verifier.md`, then run:

```bash
python3 scripts/generate.py --input <dataset.jsonl_or_csv> --source-type raw_dataset --tool-context <codex|claude|antigravity>
python3 scripts/verify.py --from-status raw_generated --source-run-id <run_id_from_generate> [--review-file <review.jsonl>]
python3 scripts/dedup.py --from-status verified_pass --source-run-id <run_id_from_generate>
python3 scripts/export.py --format csv --split 0.0
```

Prefer the DB-backed route above so the audit remains resumable and traceable.

For intentionally adversarial security corpora, injection-tolerant import is now the default. Add `--enforce-security-flags` only when you want strict flagging on those records.

### 3. `dataset audit`

Use this when the user wants a structured quality assessment of an existing or freshly generated dataset.

Read `sub-skills/dataset-auditor.md`. The auditor runs three phases:

1. **Record-level** — delegates to `data-verifier`, `deduplicator`, and optionally `llm-judge`
2. **Corpus-level** — checks split disjointness, taxonomy coverage, and context leakage
3. **Structured report** — emits a severity-classified findings table with concrete recommendations

No additional scripts are required — the auditor drives the existing `verify.py`, `dedup.py`, and `export.py` scripts and reasons over their outputs.

### 4. `dataset export`

Use this when the verified data already exists in SQLite and the user wants a specific output shape.

Read `sub-skills/formatter-exporter.md` if the schema is not obvious.

Preset export:

```bash
python3 scripts/export.py --format openai --split 0.1
```

Custom flat export:

```bash
python3 scripts/export.py --format csv --schema-file <custom_schema.json> --split 0.1
```

The flat schema file must validate before export. If the user wants custom headers, start from `resources/templates/custom_flat_schema.json` instead of inventing an ad hoc file shape.

## Natural-language prompt examples

Users do not need to use explicit flags if they describe the task naturally.

- `Generate a medical triage dataset`
- `Generate a 2000-example customer-support dataset in OpenAI JSONL`
- `Turn these URLs into a structured dataset for fine-tuning`
- `Use web research to build a fintech FAQ dataset`
- `Normalize this CSV into HuggingFace chat format`
- `Verify and clean this dataset, then export it with custom CSV headers`

## Reference files

- `sub-skills/dataset-strategy.md`
- `sub-skills/seed-generator.md`
- `sub-skills/diversity-engine.md`
- `sub-skills/dpo-pair-generator.md`
- `sub-skills/quality-filter.md`
- `sub-skills/llm-judge.md`
- `sub-skills/deduplicator.md`
- `sub-skills/formatter-exporter.md`
- `sub-skills/data-card.md`
- `sub-skills/data-verifier.md`
- `sub-skills/dataset-auditor.md`
- `sub-skills/local-collector.md`
- `resources/references/llm-audit-rubric.md`
- `resources/references/export-schema-pattern.md`

## Research/evidence route

For internet-research dataset building, use `sub-skills/research-planner.md` before `seed-generator` whenever browsing/search is available or the user asks for real-world grounding.

Recommended command:

```bash
python3 scripts/research.py --query "<topic>" --plan-file <coverage_plan.json> --tool-context <codex|claude|antigravity>
```

Then draft canonical records from `evidence.jsonl`. Real-world records should include `metadata.evidence_ids`, `metadata.reference_urls`, `metadata.source_domain`, `metadata.source_quality_score`, and `source_uri`. Raw `status: collected` chunks are not valid training examples.

- DPO plan keys (`dpo.min_pair_count`, `dpo.forbid_refusal_in_rejected`, etc.) can be added to the coverage plan to enforce contrastive quality gates.
- `review_requirements.min_capability_delta_score` and `review_requirements.require_grounding_pass` enforce structured review thresholds during verification.
- Records drafted from `evidence.jsonl` should copy `metadata.scenario_fingerprint` to prevent train/test split leakage.