Unified Ameriflux Extractor

---
name: unified-ameriflux-extractor
description: Run the combined AmeriFlux to EcoSIM extraction workflow for site metadata, NADP or tDEP atmospheric chemistry, gSSURGO soil data, and optional climate or grid NetCDF forcing. Use when building a complete site-specific EcoSIM input package.
---

# Unified AmeriFlux Extractor

## Use When

- You need a complete EcoSIM input package for an AmeriFlux site.
- You want one workflow to combine site metadata, atmospheric chemistry, tDEP deposition, gSSURGO soil profile data, and optional NetCDF forcing files.
- You need derivation reports for climate or grid variables left at fill/default values.

## Constraints
- Do not extract climate data beyond site metadata (the script only extracts site metadata, NADP, tDEP, and gSSURGO soil data).
- Site metadata extraction must use the structured `ameriflux-site-info` workflow; it should not require a local vision model.
- EcoSIM grid NetCDF files generated by this workflow do not need the optional
  atmospheric gas scalar variables `OXYEG` (or typo `OXYGE`), `Z2GEG`,
  `CO2EIG`, `CH4EG`, `Z2OEG`, or `ZNH3EG`. Do not report their absence as a
  grid derivation gap.

## Workflow

1. Resolve site metadata with `ameriflux-site-info`.
2. Extract NADP and tDEP chemistry with `ameriflux-atmchem-info`.
3. Extract soil profile data with `ameriflux-surgo-grid-extract` when gSSURGO inputs are provided.
4. Optionally invoke `Tools/create_ecosim_climate_forcing.py` or `Tools/create_ecosim_grid_forcing.py`.
5. When grid, PFT-management, or soil-management NetCDF files are created or found, also create editable Excel sidecars under `result/<SITE_ID>/`:
   - `result/<SITE_ID>/<SITE_ID>_ecosim_grid.xlsx`
   - `result/<SITE_ID>/<SITE_ID>_pft_mgmt.xlsx`
   - `result/<SITE_ID>/<SITE_ID>_soil_mgmt.xlsx`
6. When climate, grid, plant-management, and soil-management files are available, create a site namelist with `ameriflux-namelist-generator`.
7. Write merged JSON and site-specific outputs under `result/<SITE_ID>/`.

## Purpose
Combine multiple AmeriFlux data extraction capabilities into a single workflow:
1. Retrieve site metadata (latitude, longitude, elevation, mean annual temperature, Koppen climate code, IGBP vegetation type) from cached EcoSIM site JSON or structured AmeriFlux site page text.
2. Extract atmospheric chemistry (NADP) data for a range of years.
3. Extract atmospheric deposition (tDEP) data for a range of years.
4. Extract a dominant-component soil profile from a gSSURGO geodatabase using the `ameriflux-surgo-grid-extract` skill.

The result is a single JSON file containing all extracted information.

## Implementation Details
- Uses the structured extraction logic from `.agents/skills/ameriflux-site-info`.
- Reuses NADP and tDEP extraction code from `.agents/skills/ameriflux-atmchem-info`.
- Calls the gSSURGO extraction script (`extract_gssurgo_profile.py`) via subprocess to obtain soil profile data.
- Coordinates are transformed as needed for each data source.
- All data are merged under top‑level keys: `site_id`, `site_metadata`, `nadp`, `tdep`, and `gssurgo`.

## Prerequisites
- Python 3.8+.
- Install required packages:
  ```bash
  pip install playwright requests rasterio pyproj pyogrio geopandas numpy pandas netCDF4
  playwright install chromium
  ```
- Site metadata extraction uses cached JSON, static AmeriFlux HTML, or rendered DOM text. No local vision service is required.
- Access to the gSSURGO geodatabase (`gSSURGO_CONUS.gdb`).

## User-Downloaded Data

Before running the full workflow, confirm the user has downloaded or can access the source data needed for the requested outputs:

- AmeriFlux BASE/FULLSET meteorological files and BADM/BIF metadata for the target site when using curated AmeriFlux forcing or management records. Store local copies under `data/` or pass explicit paths.
- NADP precipitation-chemistry rasters and EPA tDEP deposition rasters for US atmospheric chemistry and deposition inputs.
- `gSSURGO_CONUS.gdb` for US soil/grid extraction, plus any EcoSIM grid template NetCDF requested by downstream grid tools.
- ERA5 source forcing files when `--climate-output` is requested through the local climate-forcing tool. For AmeriFlux downloads, pay attention to whether the source is half-hourly `ERA5_HH` or hourly `ERA5_HR`.
- EcoSIM assets that this workflow references but does not download: PFT parameter NetCDF, atmospheric GHG NetCDF, plant-management NetCDF for existing cases, and microbial parameter files when required by the namelist.

## Usage
```bash
python .agents/skills/unified-ameriflux-extractor/unified_ameriflux_extractor.py \
    --site-id US-Ha1 \
    --nadp-input /path/to/nadp_data \
    --tdep-input /path/to/tdep_data \
    --year1 2010 --year2 2020 \
    --gssurgo-gdb /path/to/gSSURGO_CONUS.gdb \
    --gssurgo-template /path/to/template.nc \
    --gssurgo-out result/US-Ha1/gssurgo_US-Ha1.json \
    --gssurgo-extend-last \
    --output result/US-Ha1/unified_output.json \
    --climate-output result/${site_id}/${site_id}_ecosim_climate.nc \
    --grid-output result/${site_id}/${site_id}_ecosim_grid.nc \
    --create-namelist \
    --pft-file /path/to/ecosim_pftpar_YYYYMMDD.nc \
    --atm-ghg-file /path/to/fatm_hist_GHGs_1750-2025.nc \
    --climate-data-dir data \
    --result-dir result
```
- `--site-id` is optional if latitude/longitude are provided directly, but required when generating EcoSIM climate or grid forcing files.
- If `--gssurgo-gdb`, `--gssurgo-template`, and `--gssurgo-out` are omitted, the gSSURGO extraction step is skipped.
- `--climate-output` (optional) specifies where to write the EcoSIM climate forcing NetCDF file. The script will invoke the climate forcing script located at `Tools/create_ecosim_climate_forcing.py`.
- `--grid-output` (optional) specifies where to write the EcoSIM grid forcing NetCDF file. The script will invoke the grid forcing script located at `Tools/create_ecosim_grid_forcing.py`.
- `--climate-data-dir` points to the directory containing ERA5 files (default: `data`).
- `--result-dir` is the directory for intermediate results (default: `result`).
- `--create-namelist` creates `result/<SITE_ID>/<SITE_ID>.namelist` after site forcing files are available.
- `--namelist-output` can override the namelist destination; otherwise `--create-namelist` uses the standard site result directory.
- `--namelist-run-dir` writes namelist input paths relative to an EcoSIM run directory.
- `--pft-file`, `--atm-ghg-file`, `--pft-mgmt-file`, and `--micpar-file` are passed through to `ameriflux-namelist-generator`. If `--pft-mgmt-file` is omitted, the generator searches for `result/<SITE_ID>/<SITE_ID>_pft_mgmt.nc`.
- The workflow should keep editable Excel copies for review. Use `ameriflux-surgo-grid-extract/scripts/grid_netcdf_excel_bridge.py` for grid NetCDF, `ecosim-natural-plant-mgmt/scripts/plant_mgmt_excel_bridge.py` for PFT management, and `ecosim-soil-mgmt/scripts/soil_mgmt_excel_bridge.py` for soil management.

The script will create the unified JSON output, generate any requested NetCDF forcing files and namelist, and print confirmation messages. Site-specific outputs should be placed under `result/<SITE_ID>/`.

When climate or grid NetCDF files are generated, the workflow should also emit derivation report JSON files that list template variables which could not be derived from source data and were left as fill/default values.
Do not include the optional atmospheric gas scalar variables `OXYEG`/`OXYGE`,
`Z2GEG`, `CO2EIG`, `CH4EG`, `Z2OEG`, or `ZNH3EG` in those gap lists.

Namelist generation must fail loudly if grid, climate, plant-management, PFT-parameter, or atmospheric GHG inputs are unavailable. Do not set missing grid, climate, or plant-management files to `NO`; that creates an EcoSIM setup with invalid forcing or no active vegetation management. Validate the chosen PFT parameter catalog against the plant-management PFT code before using the namelist for a run.

Climate NetCDF generation must apply range-aware gap filling:
- ERA5 meteorology values outside legitimate physical bounds are masked and filled by time interpolation before hourly aggregation.
- NADP chemistry values must be finite and non-negative, with `PHRG` constrained to pH 0-14.
- Annual chemistry gaps are filled by linear interpolation with nearest-edge filling.
- If `PHRG` cannot be derived from NADP data, write `PHRG=7` and record the policy fallback in the derivation report.

**Note:** `create_ecosim_climate_forcing.py` and `create_ecosim_grid_forcing.py` have been moved to the `Tools/` directory. All internal calls have been updated accordingly.