Data Research

---
name: data-research
version: 1.0.0
description: |
  Structured data research: search sources, extract structured data,
  archive raw sources, maintain canonical tracker pages, deduplicate.
  Parameterized via YAML recipes for investor updates, donations,
  company updates, or any email-to-structured-data pipeline.
triggers:
  - "research"
  - "track"
  - "extract from email"
  - "investor updates"
  - "donations"
  - "build a tracker"
  - "data dig"
tools:
  - search
  - query
  - page_get
  - page_put
  - link
  - add_timeline_event
  - put_raw_data
mutating: true
writes_pages: true
writes_to:
  - reports/
  - recipes/
---

# Data Research

Structured research pipeline: search sources, extract structured data,
archive raw, deduplicate, update canonical trackers, backlink entities.

## Contract

One skill for any email-to-structured-data pipeline. The only differences
between tracking investor updates, expenses, and company metrics
are the **search queries**, **extraction schemas**, and **tracker page format**.
All three use the same 7-phase pipeline with parameterized recipes.

## When to Use

- User wants to track structured data from email, web, or API sources
- User says "research", "track", "extract from email", "build a tracker"
- User mentions investor updates, donations, company metrics, filings
- User wants to set up recurring data collection (with a scheduled recipe —
  see `skills/cron-scheduler/SKILL.md`)

## Phases

### Phase 1: Define Research Recipe

Ask the user what they want to track. Either:
- Pick a built-in recipe: investor-updates, expense-tracker, company-updates
- Define a custom recipe with: source queries, classification rules, extraction schema,
  tracker page path, tracker format

Recipes are brain pages at `recipes/{name}.md` (YAML in a fenced block,
read with `page_get`). Scaffold a new one by copying a built-in recipe
page and editing its fields via `page_put`.

### Phase 2: Search Sources

Brain first (maybe we already have this data — `search`, then `query`
for structured filters). Then:
- **Email** via the agent's own email tooling: windowed queries (quarterly, monthly if truncated)
- **Web** via the agent's own web search tooling: public filings, press releases, regulatory data
- **APIs**: any structured data source the recipe defines
- **Attachments**: PDF extraction, HTML stripping

### Phase 3: Classify

Deterministic first (regex patterns from recipe), LLM fallback (Haiku
utility tier). Log every LLM fallback for future regex improvement
(fail-improve loop). Skip marketing, newsletters, noise based on the
recipe's classification rules.

### Phase 4: Extract Structured Data

**EXTRACTION INTEGRITY RULE:**
1. Save raw source immediately (before any extraction)
2. Extract fields using deterministic regex first, LLM fallback
3. When summarizing batch results: **re-read from the saved raw records**
4. Never trust LLM working memory after batch processing

This prevents a known hallucination bug where batch-processed amounts were
13/13 wrong from LLM working memory while saved records were correct.

### Phase 5: Archive Raw Sources

- `put_raw_data` for email bodies, API responses, and extracted
  attachment text (PDF/HTML content goes in as extracted text, keyed
  per source)
- Every tracker entry must link back to its raw source
  (retrieve with `get_raw_data` when re-verifying)

### Phase 6: Deduplicate

Before adding to tracker:
- Exact match (same key fields) → skip
- Fuzzy match (same entity + date + similar amount within tolerance) → flag for review
- Different amount for same entity+date → add with note (could be a correction)

### Phase 7: Update Canonical Tracker + Backlink

- Parse existing tracker page (`page_get`, markdown table)
- Append new entries in correct section (grouped by year/quarter/entity)
- Compute running totals
- Backlink every mentioned entity with `link` (person → `people/` page,
  company → `companies/` page), and record notable events with
  `add_timeline_event`
- Entity pages are enriched per the filing conventions

## Built-In Recipes

Three example recipes ship with the skill layer (see `recipes/` pages):

1. **investor-updates** — extract MRR, ARR, growth, burn, runway, headcount from investor update emails
2. **expense-tracker** — extract amounts, recipients, platforms from receipt emails (subscriptions, services, recurring charges)
3. **company-updates** — extract revenue, users, key metrics from portfolio company update emails

## Anti-Patterns

- Trusting LLM working memory for amounts after batch processing (use extraction integrity rule)
- Creating tracker entries without raw source links
- Running without deduplication (leads to double-counted entries)
- Hardcoding source-specific patterns in the pipeline code (use recipes)

## Output Format

Brain page at the recipe's `tracker_page` path with markdown tables:

```markdown
### 2026

| Date | Company | MRR | ARR | Growth | Status |
|------|---------|-----|-----|--------|--------|
| 2026-04-01 | Example Co | $188K | $2.3M | +14.7% MoM | [Source](link) |
```

Each entry links to its raw source. Running totals at the bottom of each section.

## Conventions

References conventions/quality.md (via `get_skill conventions/quality`)
for citation and back-linking rules.