Indexing

---
name: indexing
description: "Guide Kalshi and Polymarket market, trade, block, and legacy
  backfill workflows."
metadata:
  disco-role: operating
disable-model-invocation: true
license: MIT
---

# indexing

Use this sub-skill for data-collection workflows that write Parquet chunks and resume from cursor files.
It covers both API-based and blockchain-based collectors.

## Use this route when

- The user wants to backfill markets, trades, blocks, or legacy FPMM data.
- The user wants to resume a partially completed run.
- The user wants to understand the shape of a cursor file or chunked Parquet output.
- The user wants to add a new indexer subclass.

## Scope

### Included

- Kalshi market and trade backfills.
- Polymarket market, trade, block, and legacy FPMM backfills.
- Cursor files, chunk naming, deduplication, and resume behavior.
- `KalshiClient`, `PolymarketClient`, `PolygonClient`, and `Indexer.load()`.

### Excluded

- Analysis and chart output workflows, which belong to `sub-skills/analysis/`.
- Dataset packaging and host-tool installation, which belong to `sub-skills/data-ops/`.

## Read first

- `../../references/indexing-catalog.md` for the full indexer list.
- `../../references/api-reference.md` for the client and base-class APIs.
- `../../references/data-layout.md` for chunk paths, cursor files, and helper files.
- `../../references/troubleshooting.md` for cross-cutting failures.

## Core workflow

1. Identify the indexer name and required inputs.
2. Confirm the cursor file and output directory state.
3. Provide any required env vars, especially `POLYGON_RPC` for blockchain-backed collectors.
4. Run the indexer through `uv run main.py index` or directly from the Python class.
5. Verify that the expected Parquet chunks and cursor files were updated.

## Indexer families

### Kalshi

- `kalshi_markets` backfills market snapshots into chunked Parquet files.
- `kalshi_trades` fetches trades for markets discovered from the market parquet tree and skips already-processed tickers.

Important details:

- Markets use `ParquetStorage` for chunking and deduplication.
- Trades run concurrently and dedupe by `trade_id`.
- `kalshi_trades` only considers markets above the built-in volume threshold.

### Polymarket markets and trades

- `polymarket_markets` paginates the Gamma API with an offset cursor.
- `polymarket_trades` backfills CTF and NegRisk `OrderFilled` logs by block range.

Important details:

- The trade indexer stores progress after each completed block chunk.
- The trade indexer writes chunked Parquet files under `data/polymarket/trades/`.
- The market indexer writes an offset cursor and chunked market files.

### Polymarket blocks

- `polymarket_blocks` builds a block-to-timestamp lookup.
- It samples every 100 blocks and interpolates the values to fill each bucket.
- The output feeds analyses that need approximate timestamps for trade blocks.

### Polymarket legacy FPMM trades

- `polymarket_fpmm_trades` decodes `FPMMBuy` and `FPMMSell` logs from the Polygon chain.
- It filters to USDC-collateralized markets.
- It is the most RPC-intensive collector in the repo.

## Adding or fixing an indexer

- Subclass `Indexer`.
- Set `name` and `description` in `__init__`.
- Keep the fetch/resume/save loop explicit so interruption points are easy to reason about.
- Write Parquet chunks with predictable names.
- Save or update a cursor file only after the relevant unit of work is safely completed.
- Prefer small helper methods for decoding logs, chunking ranges, or deduplicating existing rows.

## Common failure patterns

- Missing `POLYGON_RPC` for chain-backed collectors.
- Invalid cursor files that point to the wrong offset or block.
- Duplicate rows when the collector is rerun without deduplication.
- Chunk files that are partially written because a run was interrupted.
- Slow or failing RPC queries when the block range is too large.

## Helpful helper

- `../../scripts/catalog.py --indexers` lists the indexer names without opening the interactive menu when run through the repo environment.