Dataset Ingestion Retrieval

---
name: dataset-ingestion-retrieval
description: "Work on RAGFlow dataset/document/chunk ingestion, parser_config,
  task execution, retrieval, RAPTOR, and GraphRAG behavior."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Dataset Ingestion Retrieval

Use this sub-skill when changing or debugging RAGFlow's dataset, document, chunk, parser configuration, indexing task, retrieval, RAPTOR, or GraphRAG behavior.

## Start Here

1. Identify the seam: REST request handling, parser configuration, task execution, document engine indexing, retrieval ranking, or graph/RAPTOR augmentation.
2. Check the relevant reference before editing:
   - `references/ingestion-pipeline.md` for built-in chunking, ingestion pipelines, task routes, embedding/indexing, RAPTOR, and GraphRAG execution.
   - `references/retrieval-and-graphrag.md` for hybrid search, metadata filters, child/TOC retrieval, KG retrieval, page rank, and retrieval response mapping.
   - `references/data-formats.md` for request fields, parser_config keys, chunk fields, metadata condition shape, and index fields.
   - `references/troubleshooting.md` for common no-chunk, stuck-task, model, queue, RAPTOR, and GraphRAG failures.
3. For parser_config changes, run the bundled offline helper on example JSON before touching service code:
   - `python scripts/inspect_parser_config.py --chunk-method naive --config parser_config.json`
   - `python scripts/inspect_parser_config.py --document-name example.pdf --config parser_config.json`

## Scope Boundaries

- Include dataset/document/chunk lifecycle, `parser_config`, ingestion dataflow, task executor behavior, `Dealer` retrieval, GraphRAG/RAPTOR, metadata filters, and API handoffs.
- Exclude DeepDoc parser internals; use the document parsing sub-skill for parser implementation details.
- Exclude public SDK client recipes; use the SDK/HTTP integration sub-skill for end-user client examples.
- Exclude deployment/service startup; use the deployment/configuration sub-skill for Redis, NATS, MySQL, MinIO, Elasticsearch/Infinity startup.
- Frontend form changes should stay minimal here and cross-check the frontend integration sub-skill.

## High-Value Checks

- Preserve the alias contract: public `chunk_method` maps to internal `parser_id`, and public `embedding_model` maps to internal `embd_id`.
- Keep dataset and document parser config merge semantics intact: dataset updates deep-merge config, document updates merge ext fields into the document config, and reparse paths may apply KB metadata config.
- For retrieval fixes, verify both `/retrieval` and dataset search endpoints when behavior is shared.
- For GraphRAG/RAPTOR changes, verify task routing, cleanup/resume behavior, and document-engine field names, not just prompt/config defaults.

## Native Verification Candidates

Prefer safe, focused tests around metadata filters, search pagination, rank feature scores, GraphRAG checkpoints/phase markers, and REST task routes for dataset/document/chunk/retrieval behavior.

## Difficult Usability Cases

- Diagnose retrieval returning no chunks even though documents show parsed, by tracing document status, doc engine index existence, embedding vector dimensions, dataset/document filters, metadata conditions, and hybrid score thresholds.
- Add a new chunk method end-to-end by tracing parser_config defaults, REST validation, task executor parser selection, frontend form alignment, and SDK/API documentation impact.