Warehouse Source New Version

---
name: warehouse-source-new-version
description: Add support for a new vendor API version to an existing Data warehouse import source, or deprecate an old one. Use when a vendor ships a new API version (Stripe date versions, Shopify quarterly versions, header-pinned revisions, /vN/ URL bumps), when implementing a version-update or deprecation task for a source under products/warehouse_sources/backend/temporal/data_imports/sources, or when repinning an ExternalDataSource to a different version. Covers deciding whether a newly announced version needs supporting at all, version declaration, dispatch, pinning semantics, deprecation metadata, and migration scripts.
---

# Adding a new vendor API version to a warehouse source

Use this skill when a vendor has released a new API version and an existing source under
`products/warehouse_sources/backend/temporal/data_imports/sources/<dir>/` must support it
**while keeping every previously supported version functional**.

## How versioning works

- Every source class (subclass of `_BaseSource` in `sources/common/base.py`) declares:
  - `supported_versions: tuple[str, ...]` — opaque vendor labels, never parsed or ordered by the framework. Default `("v1",)` (`UNVERSIONED_API_VERSION`) for vendors without meaningful versioning.
  - `default_version: str` — used when a source instance has no pin, and stamped onto newly created sources.
  - `api_docs_url: str | None` — the vendor's API docs/changelog page (where new versions are announced). Distinct from `docsUrl` (posthog.com).
  - `deprecated_versions: tuple[VersionDeprecation, ...]` — versions the vendor has deprecated (`VersionDeprecation(version=..., sunset_at=date | None)` from `sources/common/base.py`).
- Each `ExternalDataSource` row pins one version in its `api_version` column (NULL resolves to `default_version`). A schema may additionally carry a user-managed override in `ExternalDataSchema.api_version` (set from the schema's configuration page; not available for webhook-sync schemas) which wins over the source pin for that schema only. The sync pipeline resolves override → pin → default in `workflow_activities/import_data_sync.py` and hands the result to the source as `SourceInputs.api_version` — already resolved, never None there.
- **A pinned source uses its version everywhere, not just at sync time.** Every vendor-touching surface on the source classes takes an `api_version: str | None = None` parameter carrying the source instance's resolved pin (`None` → `default_version`): `get_schemas`, `validate_credentials`, `get_endpoint_permissions`, and the `WebhookSource` management methods (`create_webhook`, `sync_webhook_events`, `webhook_inputs_updated`, `get_external_webhook_info`, `delete_webhook`). Callers with a source row (creation, `refresh_schemas`, background `sync_new_schemas`, webhook endpoints, schema-scoped probes) pass the resolved pin; pre-creation flows (wizard `database_schema`, one-shot `setup`) omit it, which resolves to `default_version` — the version the new row is stamped with (`get_endpoint_permissions` currently has only the pre-creation caller, so its parameter is always `None` today). Base-path/URL/header construction from it happens inside each source. Deliberately NOT version-threaded (pure mappings or version-independent surfaces — thread them if a real vendor version ever diverges there): `get_desired_webhook_events`/`webhook_resource_map` (event-name mappings), `get_connection_metadata`, and GitHub's per-repo webhook helpers in `github_warehouse_repos.py`.
- These declarations are exposed publicly via `GET /api/public_source_configs/` (`versions`, `defaultVersion`, `apiDocsUrl`, `deprecatedVersions`) and per-instance via the source API (`api_version`, `api_version_deprecation`). The `api_version` pin is queryable in HogQL via the `data_warehouse_sources` system table.
- Registry-wide invariants are enforced by `sources/tests/test_source_versions.py`: default in supported and always the last entry (declare `supported_versions` oldest→newest; flip the default in the same PR), deprecated ⊆ supported, default never deprecated, https `api_docs_url`.

## First: does the version need to exist at all?

Spotting a new vendor label is not a reason to support it. Before touching any source file, diff the new version against the one it supersedes — the source's current `default_version`, not every entry in `supported_versions` — from the vendor's docs and changelog, area by area. Diff against the wire the client actually sends, not against the label: the `UNVERSIONED_API_VERSION` default (`v1`) does not mean the client targets the vendor's oldest API — a source built before it declared versions may already speak a modern wire, in which case adding that wire's real label is a declaration-only relabel-and-repin (below), not a new request path.

- authentication — credential fields, token/header scheme, scopes, permission probes
- base URL, version header, and the paths actually served per resource
- pagination — mechanism, params, cursor semantics, page limits
- the schema list — which endpoints/tables the source exposes
- schema formats — columns, types/formats, primary keys, incremental fields
- webhook payloads and subscription registration, for a `WebhookSource`
- rate limits, error signatures, and anything else the source's request layer touches

**If none of that differs for what this source reads, don't add the version.** Leave `supported_versions` and `default_version` untouched and close the task with the per-area, changelog-cited evidence that the new label is indistinguishable from the default here. An extra label buys nothing and costs: a pin users can select, a version the tests, API, and UI carry forever, and the implied claim that the framework dispatches on it.

Add it when any of these hold:

- **any** area above diverges from that baseline, however cosmetic it looks for our reads — then branch it (step 3). Divergence from an older still-supported label doesn't count: those pins keep serving their own request path either way (step 4), so a new label that matches the default is redundant no matter how far it sits from the legacy one;
- the vendor is retiring a version rows are still pinned to, so that label stops working — adopting the new one is the point even if the wire is identical, and the retired version moves to `deprecated_versions` in the same PR;
- the source must send the label to get the behavior it already wants (a required header or URL segment), i.e. the version is a request input, not just a name.

"Nothing changed" needs the same docs evidence as a divergence. An unread changelog is not a clean diff.

## Adding a new version, step by step

1. **Read the vendor's changelog** (the source's `api_docs_url`) and list what changed between the currently supported version(s) and the new one: renamed/removed fields, changed pagination, new required headers, changed webhook payloads, or a field the source reads becoming opt-in behind a new query parameter (a field returned by default in the old version now empty unless requested — restore it by adding that parameter on the new version's request path). Verification is docs-only — there are no stored credentials and no live-sync harness, so the docs are the sole source of truth for what each version serves. This is also the evidence the gate above runs on.
2. **Declare the version** (only once the gate says the version has to exist): add the new label to `supported_versions` and flip `default_version` to it — new sources always start on the newest stable version. A pinned row's **sync** path is unaffected by a default flip (that is the point of pinning), but two things still follow the new default: discovery/`get_schemas` if the pin isn't threaded there (step 3), and any row whose `api_version` is NULL. Reference the request layer's version constants instead of duplicating string literals.
3. **Dispatch on `SourceInputs.api_version`** at the request layer:
   - Keep it minimal. If the version is just a header/URL segment and response shapes are compatible, thread the version string down to where the client/URL is built (see Stripe: `StripeSource.source_for_pipeline` passes `self.resolve_api_version(inputs.api_version)` → `stripe_source(...)` → `StripeClient(stripe_version=...)`). Resolve through `resolve_api_version` at the source class — never hardcode a fallback version in the request layer.
   - Only introduce per-version modules/branches where behavior genuinely diverges (different pagination, different field mapping). Keep all version branching inside the source's own directory — never in shared layers.
   - A source can front several vendor API families versioned on independent tracks, so a source-level bump may move only a subset of its endpoints (and change auth for just those) — dispatch path and auth per endpoint against the resolved pin, leave the rest on their existing wire, and keep the table set identical, rather than rewriting every endpoint to the new label. The verifiable subset can also be what bounds the move: when the new version's wire is documented (paths, auth, response shape) for only some of the source's endpoints, move those to the new wire and leave the rest on their still-served legacy wire under the new default — the table set stays identical, and no endpoint ships a guessed shape. Blocking the whole bump because part of the wire is undocumented is the wrong call when the vendor still serves the old paths.
   - When the new version renames endpoints, changes primary keys, or reshapes responses, the divergence must actually be branched — never leave the old single-version request path serving the new default. All the relevant surfaces can vary by version: `get_rows` receives the resolved pin in `inputs.api_version`; credential fields can key off `default_version`.
   - Conversely, don't add inert scaffolding: an `api_version` param no caller varies, or a version→URL map with identical values, is a review finding, not forward-compat. Declaration-only (`supported_versions`/`default_version` and nothing else) is the correct shape just when the gate above passed on a non-wire reason — the old label is being retired, the vendor switches behavior account-side rather than per request, or the source already reads the vendor's newest generation under the framework's legacy unversioned label (verify the request paths the source actually builds, not the label — a legacy label can already ride the new wire, so the new label just formalizes it for new rows and both resolve identically). If the gate passed on nothing at all, there is no PR. When no-header requests resolve to a version bound to the credential account-side (not a moving "latest"), threading a version header is not merely inert — it overrides the customer's chosen version, the silent move this framework exists to prevent — so stay declaration-only and don't send one.
   - **Discovery and probe paths receive the pin — consume it.** The framework passes the resolved pin as the `api_version` parameter of `get_schemas`, `validate_credentials`, `get_endpoint_permissions`, and the webhook management methods. A multi-version source MUST build its discovery/probe/webhook clients from that parameter, not from `default_version` or a hardcoded header — otherwise a pinned source discovers/reconciles under the wrong version and its tables can disappear, duplicate, or fail reconciliation. Resolve it with `self.resolve_api_version(api_version)` — callers with a row pass an already-resolved value (mirroring `SourceInputs.api_version`), so the source-side resolve only covers pre-creation calls that pass `None`. Ignoring the parameter is only correct when you can state why the version makes no difference to that path.
   - Watch for version-dependent column hints/schemas: e.g. Stripe's `external_table_definitions` were built for specific versions. When adding a version whose response shapes differ, gate the canonical column hints to the versions they were built for and let newer versions auto-infer the schema from the data (a set of hint-compatible versions checked where hints are applied). For `has_managed_hogql_schema=True` sources this includes the read path: `hogql_definition`'s canonical column mapping is version-blind, so renamed columns need the canonical schema/descriptions updated too.
4. **Keep old versions working**: do not delete or alter the request path for previously supported versions. Removing a version is an explicit future decision, not part of a version-add PR.
5. **Tests**: extend the source's tests so both the old and new versions are exercised — at minimum that the version label reaches the client/request layer for each supported version (mock the boundary; parameterize over versions). The registry invariant test picks up declaration mistakes automatically. Don't re-test the base-class `resolve_api_version` contract (`test_source_versions.py` covers every source). When versions diverge, shape fixtures per version from the vendor docs — a v1-shaped mock under a v2 pin proves nothing.
6. **One PR per source.** Conventional title: `feat(warehouse_sources): support <vendor> API version <label>` — the scope is always `warehouse_sources` (the product), never the source dir/vendor name.

## Deprecating a version

1. Implement the newer version first (steps above) if not already supported. When the version being sunset is the vendor's **terminal** version (the whole product is discontinued and there is no successor to add), per-version deprecation cannot apply — the framework never deprecates a source's sole/default version, and there is no version to repin to. Don't fabricate a successor or relax the invariant to force the metadata: leave `supported_versions`/`default_version` unchanged, document the vendor sunset and the customer off-ramp (move off the source entirely before shutdown) in the PR, and surface whether a source-level whole-product deprecation surface is warranted for a human to decide.
2. Add the old version to `deprecated_versions` with the vendor's announced sunset date (or `sunset_at=None` if none). Never deprecate `default_version` — flip the default to the new version in the same PR.
3. The in-product warning banner and API fields light up automatically from the metadata — zero per-source UI work.
4. **Deprecated is not migrated.** Existing pins move only when the vendor has announced the version will stop being served (a sunset/removal date). A deprecation without a sunset date is advisory: mark it, leave every existing pin on it fully supported, and write no migration — repinning working customers off a version the vendor still serves is exactly the silent version move the pinning framework exists to prevent. Two narrow cases still repin under an advisory (`sunset_at=None`) deprecation: the deprecated label resolves to a byte-identical request as the new default (a pure alias — no per-version dispatch — so the repin is not a move), or the vendor already errors on the old version (e.g. `410`/`406`) so leaving pins is worse than moving them. A source that sends a per-version header/URL for a version the vendor still serves is neither — stay advisory.
5. Only for a sunsetting version: include a **written-not-run** migration script that repins affected `ExternalDataSource` rows (`api_version` column) from the deprecated version to the new one, plus any safe data/schema transforms. It must be idempotent and reviewable, and its reverse must be a no-op — repinned rows are indistinguishable from natively-created ones, so a blanket downgrade would clobber legitimate native pins. Repin only rows explicitly on the deprecated version: when the default is already the target (you are deprecating an old label without flipping the default in this PR), do not also repin NULL pins — they already resolve to the current default, so touching them is redundant churn — and leave other still-served deprecated versions alone. Where migration is lossy or unsafe — including when the new version needs credentials that can't be derived from the stored ones — do not script it: document the manual path in the PR. When only a subset of rows is unsafe (the version diverges for one endpoint that needs a credential others don't), split the cohort: script the repin for the safe rows and exclude the unsafe ones by querying their schemas, documenting only that cohort as manual — don't drop the whole migration to manual because one endpoint can't move. Do not execute migrations or backfills; humans review and run them.
6. **Never touch `ExternalDataSchema.api_version` overrides** in migration scripts — they are user-managed by design. The schema-level deprecation warning covers them; the user migrates them from the schema's configuration page.

## Pinning semantics (do not break these)

- `source.resolve_api_version(pinned)` honors a present pin verbatim — even one no longer declared — because silently moving a customer to another version is the failure mode this framework prevents. Empty string / NULL fall back to the source class's own `default_version`.
- The API create path (`_create_external_data_source` in `products/warehouse_sources/backend/presentation/views/external_data_source.py`) stamps `default_version`, and migration `0075_backfill_externaldatasource_api_version` backfilled pre-existing rows — so most rows carry a concrete pin. But `api_version` is nullable and direct-ORM creation paths that bypass the stamping (e.g. `seed_engineering_analytics.py`, and any future seeder/backfill/script) can leave it NULL, and a NULL pin resolves to `default_version` — so it follows a flip. Don't blanket-claim "every row is pinned, so a flip is safe"; verify the actual pin state for the source, and if a NULL cohort can exist, either back it out (written-not-run migration) or confirm the versions are request-identical.
- Repinning a customer = updating `ExternalDataSource.api_version` (support runbook: "Updating a warehouse source to a new vendor API version" in the PostHog/runbooks repo).

## Common pitfalls

- Vendor version labels are opaque: `"2026-02-25.clover"`, `"v21.0"`, `"2022-06-28"`. Copy them exactly; never normalize, sort, or parse.
- A source's per-endpoint URL versions (a hardcoded `/v2/...`, `/v3/...` in the endpoint config) are independent of the framework's source-level version label. A source may already call the vendor's newest per-resource routes while still carrying the `UNVERSIONED_API_VERSION` default — so a version-add can be correct as declaration-only even when the vendor's own version numbers look far apart. Diff what the source actually requests, not the vendor's headline version.
- When a vendor selects versions with a per-endpoint query param and _requires_ it on some endpoints regardless of version, those endpoints must send the selector under every pin, the legacy one included — they are version-independent even though the selector value matches the new label, and gating them on the resolved pin breaks the old pin. Gate on the pin only endpoints where the selector is optional and merely enriches the response (extra fields under the new version); those are the genuine divergence.
- A change the vendor calls "breaking" (e.g. resource ids migrating int→string) still needs no per-version branch when the source only passes the affected values through opaquely — a primary key whose _column name_ is stable (type auto-inferred), cursors forwarded verbatim. The version still has to exist (the gate passed on a real divergence), but branch the request path only where the change hits a surface you hardcode: column hints, a parsed cursor, a typed primary key.
- A version bump often changes **webhook payloads** too — if the source is a `WebhookSource`, check whether webhook-created clients (created at source-setup time, not sync time) also need the version and whether existing webhook subscriptions must be updated.
- Credential-validation paths (`validate_credentials`, permission probes) run at creation time with no row pin; they may use the default/legacy version. Changing them is optional per version bump — verify the vendor accepts the validation calls under the new version before switching them.
- A passing credential probe is not evidence sync works — the probe hits one endpoint, `get_rows` hits the rest; when they diverge per version, the probe passes while every table 404s.
- Version → header/path maps must cover every supported label — a `.get()` fallthrough silently sends no version header (tracking "latest", the drift this framework prevents). Assert coverage or raise.
- First-time versioning of a source that sends no version selector today: keep the pre-existing label (the `UNVERSIONED_API_VERSION` default) sending nothing, and add the selector only for the new dated label. That preserves already-pinned rows byte-for-byte, and pinning the new default is the point — the no-selector path was tracking the vendor account's configured version, which is the drift. This is not the fallthrough bug above: the empty selector here is deliberate and belongs to one specific legacy label, not a `.get()` miss.
- Parallel version-bump PRs grab the same next migration number; the second to merge becomes a conflicting leaf and `ci:preflight` blocks it. Check `max_migration.txt` and renumber.
- Don't regenerate schemas for existing customers as part of a version add; schema changes only apply to rows repinned via the (human-run) migration.
- Discovery diffs under the SOURCE pin (`sync_new_schemas`, `refresh_schemas`, bulk sync-defaults). A schema-level `api_version` override on a version whose table set differs from the source's version can be disabled/soft-deleted by that diff — keep overrides to short verification windows, not as a long-term way to hold one table on another version.
- When a versioned source sends the pin on any vendor call — discovery _or_ sync (a static endpoint catalog doesn't consume it at discovery, but `get_rows` still sends the version header) — add the vendor's version-rejection error signature (e.g. `406`/`410`) to `get_non_retryable_errors`, otherwise a retired pin turns the retry cadence into a permanent error loop with no user-facing surface.
- A version bump can change the auth scheme, not just the wire format. Then the source config needs both credential shapes as optional fields, auth construction dispatches on the resolved version, and `validate_credentials` enforces the pair that version needs — form-level `required` can't express "depends on the pin".
- A version that changes the pagination _mechanism_ (e.g. offset→cursor) also changes the resumable checkpoint's persisted shape. Add a new defaulted field to the resume-config dataclass rather than repurposing the existing one — so state persisted under the old version still decodes — and make the seed and save-checkpoint logic dispatch on the resolved version. Reusing one field across incompatible mechanisms corrupts in-flight resumes of the other version.
- When the vendor renames a collection between versions, keep the schema/table name set identical across versions and put the rename in a per-version path on the endpoint config — otherwise discovery diffs orphan the table on repin.
- When the new version has no equivalent for an old endpoint (a dropped collection, not a rename), keep that table only on the versions that serve it and let the table set differ by version — never map it to a guessed path just to keep the sets equal, since docs are the sole source of truth and an unverified path makes a new source surface a table that 404s at sync time.
- A source with no dispatch may currently read its version from a constant on a shared model (e.g. the OAuth `Integration` model) that also drives version-independent flows like OAuth token minting. Repoint only the sync request path onto the resolved pin; leave that constant, since bumping it changes those other flows' version with a blast radius beyond this source.
- The opposite holds when the new version is a genuinely different vendor product rather than a renamed collection (the old and new labels serve different endpoints and tables): give the new version its own disjoint table set, don't force the old names onto it. That divergence is safe only because sources are never repinned — so a 2.5→3.0-style move is lossy, and its migration is documented-not-scripted, exactly like a rename that changes primary keys.
- The shared REST framework only binds a resolved parent field into a child resource's URL **path**, not its query string (`_bind_path_params` raises "Resolve query params not supported yet"). If a new version scopes a fan-out child by a query param where the old version used a path segment, the dependent-resource `{"type": "resolve"}` machinery can't express it — iterate the parent ids explicitly in the source and bake each id into the child's params (or POST body) instead.

## Self-improvement

The default outcome of a PR is that this skill does not change. Edit it only for a learning that clears all three bars: it generalizes across sources, it would change what a future agent does, and it is not already stated or derivable from the sections above. Vendor changelog details, per-source dispatch chains or code paths, and test specifics never qualify — that context lives in your PR, not here.

When something clears the bar, fold it into the section where an agent would need it (the gate, a step, a pitfall) as one vendor-neutral line. Do not append a learnings list, changelog, or dated notes anywhere in this file.