Indexing And Search

---
name: indexing-and-search
description: "Build ColBERT indexes, load existing indexes, run single-query and
  batch retrieval, tune search hyperparameters, and save rankings safely."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# ColBERT Indexing and Search

Use this sub-skill when the task is to turn a TSV passage collection into a ColBERT index, open an existing ColBERT index, retrieve top-k passages for one query or many queries, tune retrieval speed/quality settings, validate collection/query/ranking inputs, or diagnose index path and ranking-output issues.

## Route the task

- For public imports, verified signatures, path resolution, return shapes, overwrite modes, and CPU/GPU behavior, read `references/api-reference.md`.
- For copy-editable indexing, existing-index loading, single-query search, batch search, ranking save, `index_root`, mmap, and tuning workflows, read `references/index-search-workflows.md`.
- For checkpoint, optional dependency, backend, TSV/config, API misuse, overwrite, `qid_to_pids`, and output-location failures, read `references/troubleshooting.md`.
- For deterministic preflight checks of user-provided collection, query, and ranking TSV files before expensive ColBERT runs, use `scripts/validate_colbert_inputs.py`.
- For an editable minimal script that exposes `index`, `search`, and `search-all` subcommands with public ColBERT APIs, adapt `scripts/minimal_index_search_template.py`.

## Core boundaries

- Cover `Indexer`, `Searcher`, `Run`, `RunConfig`, `ColBERTConfig`, `Collection`, `Queries`, `Ranking`, `index_root`, overwrite/reuse/resume, `load_index_with_mmap`, and `ncells`/`ndocs` search tuning.
- Keep training, distillation, triples, and checkpoint training workflows in `../training-and-distillation/`.
- Keep TSV/qrels/evaluation details beyond basic indexing/search I/O in `../data-and-evaluation/`.
- Keep mutable add/remove/persist index serving workflows in `../index-updates-and-serving/`.
- Keep tokenizer internals, checkpoint architecture, and model compatibility analysis in `../modeling-and-tokenization/`.

## Fast starting point

1. Validate TSV shapes before expensive work: `python scripts/validate_colbert_inputs.py --collection collection.tsv --queries queries.tsv`.
2. Index with explicit `checkpoint`, `root`, `experiment`, `index_name`, and overwrite policy.
3. Search by reusing the same `root`, `experiment`, and `index_name`, or pass explicit `index_root` when the index is outside the default layout.
4. Save batch results with `Ranking.save(...)`; use an absolute output path or print the returned path because relative paths are resolved by the active `Run` context.

## Verified package facts

- Distribution: `colbert-ai` 0.2.22; import package: `colbert`.
- Verified imports include `colbert`, `colbert.infra`, `colbert.data`, `colbert.modeling.checkpoint`, `utility`, and `baleen`.
- CPU import inspection works, but practical indexing and training usually require CUDA/GPU; CPU search is possible and mmap search is CPU-only.
- Important signatures: `Indexer(checkpoint, config=None, verbose=3)`, `Searcher(index, checkpoint=None, collection=None, config=None, index_root=None, verbose=3)`, `Collection(path=None, data=None)`, `Queries(path=None, data=None)`, and `Ranking(path=None, data=None, metrics=None, provenance=None)`.