Adding Inbox Sources

---
name: adding-inbox-sources
description: Add a new warehouse-backed source to the PostHog Desktop Self-driving Inbox (the feature that ships GitHub, Linear, Zendesk, pganalyze, Jira). A source syncs one warehouse table (issues/tickets/conversations) and a cloud "signals scout" watches it and emits findings. Use when asked to "add a new inbox/self-driving source", "wire up <Jira/GitLab/Sentry/Intercom/Freshdesk/Front/Gorgias/etc> as a signal source", or to extend the source-toggle grid. Covers all three surfaces (posthog/posthog scout emitter + posthog/code UI wiring + the context-mill self-driving wizard skill that offers the source in `npx @posthog/wizard self-driving`), the deploy ordering between them, and created_via attribution.
---

# Adding a Self-driving Inbox source

The Self-driving Inbox connects a PostHog **data-warehouse source** (GitHub, Linear,
Zendesk, pganalyze, Jira today), syncs one "actionable records" table (`issues` /
`tickets` / `conversations`), and a server-side **signals scout** in
`posthog/posthog` watches new rows and emits findings/reports into the Code Inbox.

**Adding a source touches three surfaces.** The scout is not generic over arbitrary
tables — it is driven by a static registry keyed on `(source_type, table)`. The
first two surfaces are mandatory (without them the source can't emit signals or be
toggled in the app); the third is what makes the CLI `self-driving` wizard _offer_
the source during onboarding.

This skill lives in `posthog/posthog`; the UI lives in the separate `posthog/code`
repo; the wizard's onboarding script lives in `PostHog/context-mill`:

| Surface                       | What changes                                                                                                                                                                                                                                                            |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `posthog/posthog` (this repo) | New scout emitter + registry entry + `SignalSourceProduct` enum value (+ migration) + contract variant. **The data-warehouse source itself must already exist** (all Tier-1 sources do).                                                                                |
| `posthog/code`                | ~8 UI/wiring files: the source-product unions, toggle card, setup form, hook maps, icon, filter option (+ OAuth service/router only for OAuth sources).                                                                                                                 |
| `PostHog/context-mill`        | The `self-driving` skill's connected-tools list, so `npx @posthog/wizard self-driving` offers the source. **Optional** — skip it and the source still works everywhere else; the wizard just won't proactively suggest it. See "The self-driving wizard surface" below. |

> Backend work in `posthog/posthog` should be done in a **git worktree** (see
> "Worktree setup" below). Merges in both repos go through the Trunk merge queue —
> see the `merging-prs` skill.

## Deploy ordering (do not skip)

The backend `SignalSourceProduct` choice must ship **before** the Code UI surfaces
the toggle. The Code toggle calls `createSignalSourceConfig({ source_product })`,
and the Django model rejects a `source_product` that isn't in its choices → 400.

So: **land the posthog/posthog PR(s) first (or at least the enum migration), then
the posthog/code PR.** When opening both together, note the dependency in the Code
PR description.

## The setup form: use the dynamic renderer, don't hardcode

**Do not hand-code a per-source setup form.** PostHog Cloud serves each source's
connect-form field schema over HTTP, and the app renders it generically via
`DynamicSourceSetup` (`packages/ui/src/features/inbox/components/DynamicSourceSetup.tsx`).
A new credential-based source needs **zero form code** — just route its
`DataSourceSetup` switch case to `DynamicSourceSetup` with the capitalized
`sourceType` and the `schemas` to sync.

- Endpoint: `GET /api/environments/{projectId}/external_data_sources/wizard/?source_type=<Type>` → `Record<string, SourceConfig>`. Client method: `PostHogAPIClient.getExternalDataSourceConfigs`. Hook: `useSourceConfig(sourceType)`.
- `SourceConfig.fields` is a union: `input` (text/email/password/url/number/…), `select`, `switch-group`, `oauth`, `oauth-account-select`, `ssh-tunnel`, `file-upload`. `DynamicSourceSetup` renders input/select/switch-group **and `oauth`/`oauth-account-select`** generically, and builds the `createExternalDataSource` payload from field `name`s. The backend is the single source of truth for field names/labels/required/secret, so forms never drift.
- The field `name`s become the `payload` keys — so you no longer hand-maintain them. (Jira → `subdomain`, `email`, `api_token`; all `secret:false` except the token.)

**OAuth sources need NO bespoke form.** `DynamicSourceSetup` renders the `oauth` field (a
connect button that starts the flow by `kind` and polls `getIntegrationsForProject` for the new
integration, writing its id into `payload[<name>]`) and the `oauth-account-select` field (a
server-side-searched picker over the integration's resources) generically. The connect flow is
started by the generic, `kind`-parameterized `integration` tRPC router
(`packages/host-router/src/routers/integration.router.ts` → `IntegrationService`), so any
provider in `OauthIntegration.supported_kinds` works with no per-kind service or router. So an
OAuth source (Intercom, HubSpot, Salesforce, …) is the same one-registry-entry change as a
credential source — route its `DataSourceSetup` case to `DynamicSourceSetup`.

Only two field types still lack a generic renderer (disable submit): `ssh-tunnel` and
`file-upload`. Route those to a bespoke form. Resource pickers that must run _after_ OAuth
(GitHub's repo picker) are handled by `oauth-account-select`; the old `GitHubSetup`/`ZendeskSetup`
/`PgAnalyzeSetup` hardcoded forms remain only for historical reasons — leave them or migrate to
`DynamicSourceSetup` opportunistically; don't add new ones.

Supported OAuth `kind` values (posthog `OauthIntegration.supported_kinds`,
`posthog/models/integration/oauth.py`): `slack, salesforce, hubspot, google-ads,
google-analytics, google-search-console, google-sheets, snapchat, linkedin-ads,
reddit-ads, tiktok-ads, bing-ads, meta-ads, intercom, linear, clickup, jira,
pinterest-ads, stripe` (+ `github` via App install). **A source not in this list
must use the API-key form** — its warehouse connector takes credentials directly,
independent of the OAuth integration list. (Note: even Jira's _warehouse source_
uses an API token, not the OAuth `kind=jira`.)

## Source catalog (Tier-1)

`source_type` = the capitalized DWH type string passed to `createExternalDataSource`.
Verify exact `source_type` + `payload` key names against the posthog
`external_data_sources` serializer / the source's `source.py` at implementation time.

| Product   | `source_type` | Table           | Auth                    | Setup template       | `payload` keys                                     |
| --------- | ------------- | --------------- | ----------------------- | -------------------- | -------------------------------------------------- |
| Jira      | `Jira`        | `issues`        | API token               | `DynamicSourceSetup` | `subdomain`, `email`, `api_token`                  |
| GitLab    | `GitLab`      | `issues`        | API token               | `DynamicSourceSetup` | `gitlab_host`, `personal_access_token`, `project`  |
| Sentry    | `Sentry`      | `issues`        | API token               | `DynamicSourceSetup` | `auth_token`, `organization_slug`, `api_base_url?` |
| Freshdesk | `Freshdesk`   | `tickets`       | API key                 | `DynamicSourceSetup` | `subdomain`, `api_key`                             |
| Front     | `Front`       | `conversations` | API token               | `DynamicSourceSetup` | `api_token`                                        |
| Gorgias   | `Gorgias`     | `tickets`       | API key                 | `DynamicSourceSetup` | `gorgias_domain`, `email`, `api_key`               |
| Intercom  | `Intercom`    | `conversations` | OAuth (`kind=intercom`) | `DynamicSourceSetup` | `intercom_integration_id`                          |

(Zendesk `tickets`, GitHub `issues`, Linear `issues`, pganalyze `issues`+`servers`,
and Jira `issues` are already shipped — copy them, don't re-add. The Jira row above
stays as the canonical worked example for `payload` keys and the JSON-blob emitter
gotcha below.)

---

## posthog/code checklist (per source)

Product key is the lowercase `source_product` (e.g. `"jira"`). Grep the repo for
`"zendesk"` (case-insensitive) inside `packages/` — every hit that is
source-list-relevant is a place you must add the new product. The canonical list:

### Type gates (every source)

1. `packages/shared/src/inbox-types.ts` — add `"jira"` to the `SourceProduct` union.
2. `packages/api-client/src/posthog-client.ts` — add to `SignalSourceConfig.source_product` union; add a new `source_type` value only if the record type isn't already `issue`/`ticket`.

### Live UI path (every source)

3. `packages/ui/src/features/inbox/hooks/useSignalSourceToggles.ts` — `SetupSourceProduct`, `SOURCE_TYPE_MAP`, `SOURCE_LABELS`, `DATA_WAREHOUSE_SOURCES` (`{ dwSourceType, requiredTable }`), `ALL_SOURCE_PRODUCTS`, and the `computeValues` initializer object.
4. `packages/ui/src/features/inbox/components/SignalSourceToggles.tsx` — `SignalSourceValues` field, a `toggleX`/`setupX` callback, and a `<SignalSourceToggleCard>` in the "External connections" column (icon/label/description + `requiresSetup`/`onSetup`/`loading`/`syncStatus`).
5. `packages/ui/src/features/inbox/components/DataSourceSetup.tsx` — `DataSourceType`, `REQUIRED_SCHEMAS`, and the `switch` case. For a credential source, route the case to `<DynamicSourceSetup sourceType="Jira" title="Connect Jira" schemas={schemasPayload("jira")} … />` — **no new form component**. Only OAuth/resource-picker sources need a bespoke `XSetup`.
6. `packages/ui/src/features/inbox/components/utils/source-product-icons.tsx` — `SOURCE_PRODUCT_META` entry (`Icon`, `color`, `label`). Add an inline SVG icon file (copy `PgAnalyzeIcon.tsx`) if no Phosphor icon fits.
7. `packages/ui/src/features/inbox/filterOptions.tsx` — `INBOX_SOURCE_OPTIONS` entry (source-filter dropdown).

### Core mirror (keep in sync — `SignalSourceService`/`DataSourceService` mirror the maps; not on the live UI path today but keep them consistent)

8. `packages/core/src/inbox/signalSourceService.ts` — mirror `SOURCE_TYPE_MAP`, `DATA_WAREHOUSE_SOURCES`, `ALL_SOURCE_PRODUCTS`, `computeSourceValues` init, plus `WarehouseSourceProduct`/`SignalSourceValues`.
9. `packages/core/src/inbox/dataSourceService.ts` — `DataSourceType`, `REQUIRED_SCHEMAS`, a `createXDataSource` method.

### OAuth plumbing — NOT needed per source anymore

There is now a **generic, `kind`-parameterized** integration flow: `IntegrationService`
(`packages/core/src/integrations/integration.ts`) + the `integration` tRPC router
(`packages/host-router/src/routers/integration.router.ts`). `DynamicSourceSetup` starts any
OAuth flow through it via the field's `kind`. So a new OAuth source needs **no** per-kind
service, symbol, or router — do not clone `linear.ts`/`linear-integration.router.ts` per source.
(The old per-kind linear/slack/github routers still exist for other callers; leave them.)

### Setup-form specifics

- **Credential source:** route the `DataSourceSetup` switch case to `DynamicSourceSetup`. Nothing else — the fields come from the wizard endpoint.
- **OAuth source:** also just route to `DynamicSourceSetup`. Its connect-form schema carries the
  `oauth` field (and, if the provider needs a resource picked, an `oauth-account-select` field),
  which `DynamicSourceSetup` renders generically — connect button + integration polling + account
  picker, all by `kind`. No bespoke form. The provider must be in
  `OauthIntegration.supported_kinds`.
- Issues sources (`github`/`linear`/`jira`) force `issues` to `full_refresh` in `ensureRequiredTableSyncing` (`useSignalSourceToggles.ts`) — add the new product to that condition if it syncs an `issues` table (issues get edited/closed, so incremental append would miss updates). Ticket/conversation sources only force `should_sync=true`.

### Verify

- `pnpm --filter @posthog/shared build` after touching `inbox-types.ts` (it's a published type).
- `pnpm typecheck` (whole repo — the unions are consumed across packages).
- `biome lint packages/core packages/ui` — zero `noRestrictedImports`, imports ordered.

---

## posthog/posthog checklist (per source)

Scout lives in `products/signals/backend/`. The warehouse→signals handoff is
generic; only the per-source emitter + registry entry are new.

1. **Enum** — `products/signals/backend/enums.py`: add a `SignalSourceProduct` value + a `SIGNAL_SOURCE_PRODUCT_LABELS` entry. This regenerates `SIGNAL_SOURCE_PRODUCT_CHOICES` used by the model.
2. **SourceType** — `products/signals/backend/models.py` `SignalSourceConfig.SourceType`: only add if the record type isn't already `ISSUE`/`TICKET`.
3. **Migration** — `python manage.py makemigrations signals` → `NNNN_alter_signalsourceconfig_source_product`. **Batch all new enum values into ONE migration when doing several sources** — parallel PRs each adding a migration to this model collide in the merge queue.
4. **Emitter** — new module `products/signals/backend/emission/<source>_<table>.py` exporting a `SignalSourceTableConfig`: `partition_field` (incremental cursor column), `fields` (columns to SELECT), optional `where_clause`, `partition_field_is_datetime_string`, an `emitter(row) -> SignalEmitterOutput(source_product, source_type, source_id, description, weight, extra)`, and optional LLM `actionability_prompt`/`summarization_prompt`. **Copy `github_issues.py` and adapt to the warehouse table's columns** (read the source's `settings.py`/`canonical_descriptions.py` under `products/warehouse_sources/backend/temporal/data_imports/sources/<name>/` for exact column names).

   ⚠️ **Not every source stores flat columns.** GitHub/Linear expose `title`/`body`/`created_at` as top-level columns, so the generic `data_warehouse_record_fetcher` (`SELECT {fields} FROM {table} WHERE {partition_field} > cursor`) works directly. Others do **not**: e.g. **Jira's `issues` table has only `id`, `key`, `self`, `fields` (a nested JSON blob), `expand`** — `summary`/`description`/`status`/`created` all live inside `fields`. For such sources you must SELECT `fields` and `JSONExtractString(fields, '…')` in the emitter, and the `partition_field` must be a JSON expression (`JSONExtractString(fields, 'created')`, `partition_field_is_datetime_string=True`). **Verify the generic fetcher accepts a JSON-expression `partition_field`** (it interpolates it into HogQL `WHERE`) before assuming a clone works — this is the single most likely thing to be subtly wrong, and it can only be confirmed by running a sync, not by reading code. Inspect the real column shape via the source's `canonical_descriptions.py` first.

5. **Register** — `products/signals/backend/emission/registry.py` `_register_all_emitters()`: `register((ExternalDataSourceType.X, "<table>"), <config>)`.
6. **Contract** — `products/signals/backend/contracts.py`: add a `Literal[SignalSourceProduct.X]` variant.
7. The generic fetcher (`emission/fetchers/data_warehouse.py`), the gate (`emission/gate.py`), and the warehouse hook plumbing need **no** changes.

### Verify

- Migration applies cleanly; `python manage.py makemigrations --check` is clean afterward.
- The `ExternalDataSourceType` member exists in `products/warehouse_sources/backend/types.py`.

---

## The self-driving wizard surface (`PostHog/context-mill`) — optional

The two surfaces above make a source work in the app. They do **not** make the CLI
`npx @posthog/wizard self-driving` onboarding _offer_ it. That flow is an AI agent
driven by a skill in the separate `PostHog/context-mill` repo, and its connected-tool
list is **hardcoded** — it is not read dynamically from the wizard endpoint. So a new
source is invisible to self-driving onboarding until you add it there.

Edit `context/skills/self-driving/references/5-connected-tools.md` (plus the source-list
mentions in `4-sources.md`, `2-read-context.md`, and `description.md`):

1. Add the tool to the step-5 `wizard_ask` multi-select options (`{ label, value }`).
2. Add its responder mapping — the same `source_product` / `source_type` pair as the
   posthog emitter (e.g. Jira → `jira` / `issue`).
3. Add its DWH `source_type` to the "already connected" detection list.
4. Classify how the run connects it:
   - **Credential source** (API key/token — Jira, Zendesk, pganalyze): the run can't
     collect credentials in-flight, so it never sends the user to the UI. Group it with
     Zendesk — arm the dormant responder and record a follow-up. **No connector file.**
   - **One-click OAuth source** (Linear-style, `kind=<source>` integration): add a
     dedicated connector reference (`5X-<source>.md`, clone `5b-linear.md`) that hands
     over the authorize link and creates the source from the integration id.

`dist/` is gitignored (built in CI), but run `node scripts/build.js` (Node ≥20) once to
confirm the skill bundles; the build validates structure.

## created_via attribution

`ExternalDataSource.created_via` records **how** a source was created and is derived
**server-side from the request transport** (`get_event_source` in
`posthog/event_usage.py`, keyed on the user-agent) — an API caller **cannot** set it
directly, which is deliberate (otherwise any client could self-label). A new source
inherits this for free; there is nothing per-source to wire. For reference, the machine
value `mcp` is upgraded based on the caller:

| Caller                                       | Transport                  | `created_via`  |
| -------------------------------------------- | -------------------------- | -------------- |
| PostHog Desktop app inbox (`posthog/code`)   | `EventSource.POSTHOG_CODE` | `self_driving` |
| `npx @posthog/wizard` (self-driving program) | `EventSource.WIZARD`       | `self_driving` |
| Other MCP clients                            | `EventSource.MCP`          | `mcp`          |

The upgrade lives in `_create_external_data_source` in
`products/warehouse_sources/backend/presentation/views/external_data_source.py`.

---

## Worktree setup (posthog/posthog backend work)

```bash
# from your local posthog/posthog clone
git fetch origin
git worktree add ../worktrees/inbox-<source> -b <your-branch-prefix>/inbox-<source>-source origin/master
```

If the agent sandbox has no GPG signing key configured, disable signing for the
commit (`git -c commit.gpgsign=false commit`) so it doesn't fail on a missing
key. Do **not** reflexively add `--no-verify`: the pre-commit/pre-push hooks run
lint, type-check, and `ci:preflight`, and skipping them lets breakage reach CI.
Only bypass a specific hook if it genuinely can't run in the worktree, and run
the Verify steps above by hand first. Open PRs **ready for review** (not draft).
Use semantic commit prefix `feat(data-warehouse):` for anything touching
warehouse sources / signals. Clean up the worktree when merged:
`git worktree remove ../worktrees/inbox-<source>`.

## PR conventions

- One PR per source per repo (per the request). Title: `feat(data-warehouse): add <Source> as a self-driving inbox source`.
- Base the PR body on the repo's PR template.
- Backend PRs first (or the shared enum migration first); Code PR references the backend PR and the deploy-ordering dependency.
- **Migration hygiene:** if opening several backend PRs, put all new `SignalSourceProduct` values in ONE prep PR's migration, and have the per-source emitter PRs carry no migration — otherwise they conflict.
- Follow the `merging-prs` skill to land each PR through the Trunk queue.

## Gotchas

- `source_type` (DWH, capitalized e.g. `"Jira"`) ≠ `source_product` (lowercase e.g. `"jira"`) ≠ `SourceType` record kind (`"issue"`). Three distinct fields.
- `DATA_WAREHOUSE_SOURCES` matches an existing external source by `source_type.toLowerCase()` — so `dwSourceType` must equal the real DWH type string.
- The exact `payload` keys are the real risk. They must match the posthog serializer for that `source_type`. Confirm against `source.py` before shipping.
- Keep `SOURCE_PRODUCT_META` and `INBOX_SOURCE_OPTIONS` covered for every product (the inbox `CLAUDE.md` calls this out) or findings render without an icon/filter.
- Do **not** confuse this with `packages/core/src/onboarding/githubConnectService.ts` — that's repo-access (clone/branch/PR), unrelated to warehouse sources.