Data Doc

---
name: data-doc
description: Document datasets, variables, sources, and merge keys for replication
---

# Data Documentation Assistant

Help document datasets systematically for replication packages, co-author handoffs, and future reference.

## Dataset Documentation Template

When documenting a dataset, capture:

### 1. Dataset Overview
- **Name:** (e.g., `firm_year_panel.parquet`)
- **Location:** (relative path from project root)
- **Created by:** (script that generates it)
- **Last updated:**
- **Unit of observation:** (firm-year, loan-quarter, etc.)
- **N observations:**
- **Time coverage:**

### 2. Source Data
| Source | Access | Raw File | Notes |
|--------|--------|----------|-------|
| WRDS Compustat | Subscription | `raw/compustat_funda.csv` | Annual fundamentals |
| Revelio Labs | Licensed | `raw/revelio_positions.parquet` | Via BU server |
| Hand-collected | Manual | `raw/manual_coding.xlsx` | See coding protocol |

### 3. Key Variables

| Variable | Type | Description | Source | Notes |
|----------|------|-------------|--------|-------|
| `gvkey` | str | Compustat firm identifier | Compustat | Primary key |
| `fyear` | int | Fiscal year | Compustat | |
| `at` | float | Total assets ($ millions) | Compustat | Winsorized 1/99 |
| `treated` | int | =1 if treated firm | Constructed | See section 4 |

### 4. Variable Construction

For constructed/derived variables, document:

```
Variable: treated
Definition: =1 if firm received first PE investment in year t
Construction:
  1. Merge PitchBook deals to Compustat on EIN
  2. Keep first deal per firm
  3. Flag year of first investment
Script: 2a_construct_treatment.py, lines 45-78
```

### 5. Sample Filters

Document all filters applied:

| Filter | Observations Dropped | Remaining |
|--------|---------------------|-----------|
| Raw data | - | 150,000 |
| Drop financials (SIC 6000-6999) | 25,000 | 125,000 |
| Require non-missing assets | 5,000 | 120,000 |
| Require 2+ years in panel | 10,000 | 110,000 |

### 6. Merge Keys

| Dataset A | Dataset B | Key(s) | Match Rate | Notes |
|-----------|-----------|--------|------------|-------|
| Compustat | CRSP | gvkey | 95% | Via CCM link table |
| Compustat | PitchBook | EIN | 72% | Manual cleaning needed |
| Revelio | Compustat | company_name | 68% | Fuzzy match, see script |

---

## Quick Commands

- **"document this dataset"** - Generate full template for a dataset
- **"variable list"** - Create variable table only
- **"merge documentation"** - Focus on merge keys and match rates
- **"sample flow"** - Generate sample filter table
- **"codebook"** - Formal codebook format for replication package

---

## Codebook Format (for Replication Packages)

```
================================================================================
CODEBOOK: firm_year_panel.dta
Generated: 2026-01-15
================================================================================

IDENTIFICATION
  gvkey         Compustat permanent firm identifier
  fyear         Fiscal year

OUTCOME VARIABLES
  roa           Return on assets = ni/at (winsorized 1/99)
  investment    Capex/lagged assets = capx/L.at

TREATMENT VARIABLES
  post          =1 for years after treatment
  treated       =1 for firms ever treated
  treat_post    Interaction: treated × post (DiD coefficient)

CONTROLS
  size          Log total assets = ln(at)
  leverage      Book leverage = (dltt+dlc)/at
  mtb           Market-to-book = (prcc_f×csho)/ceq

FIXED EFFECTS
  ff48          Fama-French 48 industry classification
  state         State of incorporation
================================================================================
```

---

## Data Provenance Checklist

Before finalizing a dataset, verify:

- [ ] All source files documented with access instructions
- [ ] Variable definitions are unambiguous
- [ ] Sample filters documented with observation counts
- [ ] Merge match rates reported
- [ ] Winsorization/trimming documented
- [ ] Missing value treatment documented
- [ ] Script that creates dataset is identified
- [ ] Date of creation recorded