Raven

> **MANDATORY EXECUTION POLICY** — READ BEFORE PROCEEDING
>
> You MUST run the **actual model binary or package** described in this document.
> If the model fails to import, compile, or execute, you MUST:
> 1. Check `diagnostics/triplets.yaml` for a matching error pattern
> 2. Try the fix suggested in the triplet's `remedy` section
> 3. If still failing, report the error to the user with full details
>
> You MUST NOT substitute a simplified Python formula, regression equation,
> or hand-coded approximation in place of the real model. Doing so produces
> scientifically invalid results and defeats the purpose of the KI.
>
> Before starting, run: `python preflight_check.py` (in this KI directory)
> to verify that the model binary/package and required data are available.
>
> **DEBUGGING PROTOCOL** — When something goes wrong, follow this order:
> 1. **Check triplets** — `diagnostics/triplets.yaml` may already cover this error
> 2. **Read official docs** — The model's own documentation for expected formats/units
> 3. **Find working examples** — Check `outputs/` or the model's shipped test data
> 4. **Fix the tool** — With knowledge of what "correct" looks like
>
> Do NOT write custom debug scripts. The answers are in the docs and examples.

<!-- KI-MAP:BEGIN (projected by generate_skill_map.py — edit the KI, not this table) -->
## KI map — what to read, and when

| when you need | read | why |
|---|---|---|
| FIRST, always | `preflight_check.py` | run it (`python preflight_check.py`): proves env/binary/data are usable and emits a machine-readable `PREFLIGHT_REPORT=` line. Do not debug a run that never had a healthy environment. |
| to run the pipeline stages | `tools/` (11 tools) | the executable pipeline. Read each tool's argparse (`--help`) before composing a command; SKILL.md's stage table says which tool serves which stage. |
| before running a stage | `docs/s*_*.md` (6 stage docs) | per-stage procedure, verification and traps — the how-to that SKILL.md's overview compresses. |
| on ANY error, before debugging | `diagnostics/triplets.yaml` (46 entries) | symptom → diagnosis → remedy for this model's known failure modes. Check here FIRST; the answer usually exists. Never renumber or rewrite entries. |
| to know what an output IS | `dag.yaml` | the model's identity: every output's medium, units, `validation_rank` (1 = the headline variable) and observability. Scoring and obs-binding read THIS — when asked 'what does this model predict', the dag is the answer, not a guess. |
| when building inputs / parsing outputs | `docs/format_spec.yaml` | exact I/O shapes + `known_issues`, projected from dag + triplets. Regenerate with `ki_tools_common/generate_format_spec.py` after changing either — never hand-edit. |
| to judge a run's skill | `docs/validation_convention.yaml` | how this model's field judges it validated: per-`dag_variable` metrics, directions and CITED pass-bands. A run is graded against these, not against intuition. |
| for claims and thresholds | `docs/gathered_papers.json` (27 papers) + `docs/papers_index.md` | the literature this KI is judged by; each entry's `text_path` is fetched full text in the central paper cache. `role: benchmark` marks the model's own skill paper. |
| for a machine-readable summary | `knowledge_infrastructure.yaml` | the manifest (package, pipeline, validation tier, counts) — projected by `ki_tools_common/generate_ki_manifest.py`; regenerate after structural changes, never hand-edit. |
| what past runs learned | `.kdt_evolution.jsonl` | append-only memory of previous runs and fixes on this KI. |

*Projected 2026-10-03 from the KI's actual contents — 10 components present. Refresh: `python3 ki_tools_common/generate_skill_map.py --ki_dir <this KI>`.*
<!-- KI-MAP:END -->

<!-- KI-TOOL-INDEX:BEGIN (projected by generate_skill_map.py — the discoverability contract: every public tool, exact path; PURPOSE stays human-authored elsewhere) -->
### Executable tool index (projected — complete by construction)

Every public tool in this KI, by exact path. What each is FOR lives in the
human-written Tool Inventory above; `--help` on any of these prints its arguments.

| tool (exact path) | invocation |
|---|---|
| `tools/common/validate_raven_inputs.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/common/validate_raven_inputs.py --help` |
| `tools/s0_config/select_model_template.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/s0_config/select_model_template.py --help` |
| `tools/s10_coupling/raven_vic_comparison.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/s10_coupling/raven_vic_comparison.py --help` |
| `tools/s1_basin_setup/build_rvh_from_shapefile.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/s1_basin_setup/build_rvh_from_shapefile.py --help` |
| `tools/s2_parameters/build_rvp_parameters.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/s2_parameters/build_rvp_parameters.py --help` |
| `tools/s3_forcing/convert_forcing_to_rvt.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/s3_forcing/convert_forcing_to_rvt.py --help` |
| `tools/s5_initial_conditions/generate_rvc_initial.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/s5_initial_conditions/generate_rvc_initial.py --help` |
| `tools/s6_execution/run_raven.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/s6_execution/run_raven.py --help` |
| `tools/s7_output/parse_raven_output.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/s7_output/parse_raven_output.py --help` |
| `tools/s8_ensemble/run_ensemble_comparison.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/s8_ensemble/run_ensemble_comparison.py --help` |
| `tools/s9_calibration/calibrate_raven_dds.py` | `KISSPATH_PYTHON_ENV/bin/python {KI}/tools/s9_calibration/calibrate_raven_dds.py --help` |

*11 public tools; `_`-prefixed helpers and packaging files excluded.*
<!-- KI-TOOL-INDEX:END -->

---

## Data Preparation

### Forcing data

**Data Sources**: Raven's forcing is built straight from the data source (CMFD, MSWX or NASA POWER) by this KI's
own tool, `tools/s3_forcing/convert_forcing_to_rvt.py`. The tool reads the source only through
`ki_tools_common.load_forcing` (`load_daily_forcing` for one point, `load_daily_forcing_points` for many
points in one shared pass). Name the source with `--forcing_source cmfd|mswx|nasa_power` (required) and the
place with ONE of:

| Place | Use it for |
|-------|-----------|
| `--basin_shp <polygon>` | a basin run (the normal case): mean over every source grid-cell centre inside the polygon, cos(latitude) weights |
| `--points_csv <lat,lon rows>` | the plain mean of chosen points |
| `--lat --lon` | one point — only for a basin inside one or two source cells (dt_rav_048) |

Raven needs no other model's files and no other model's run for its input. `--obs_file` writes the
observation `.rvt` the main `.rvt` redirects to; `--gauge_elev` is the elevation the series stands for (cmfd:
read from the store when not given). A missing value, an uneven time axis or a period not fully covered stops
the tool and nothing is written. Details: `docs/s3_forcing_conversion_skill.md`.

**Data Validation Reference**: See `data_ki/CMFD/SKILL.md` for CMFD unit documentation and known traps.

---

# Raven Hydrological Modelling Framework v4.1 — Knowledge Infrastructure

**Package**: `hydrocraft-raven` v1.0.0
**Model**: Raven v4.1 (University of Waterloo, Prof. James Craig)
**Created by**: Jianyun Zhang Research Group, Hohai University
**Last updated**: 2026-03-21
**Stats**: 9 error log entries | current tool, stage-doc and triplet counts: the KI map at the top of this file
**Validation Status**: `production_validated`

---

## Overview

This knowledge infrastructure enables fully autonomous construction and execution of the **Raven Hydrological Modelling Framework** on any basin. Raven is unique among HydroCraft models because it is a **meta-framework** that can emulate 15+ different hydrological model structures (GR4J, HBV, HMETS, SAC-SMA, HYMOD, UBC, etc.) from a single codebase.

**What Raven does**: Assembles custom hydrological models from a library of 120+ process algorithms. Each hydrological process (infiltration, ET, snowmelt, baseflow, routing) is an interchangeable module. Users select algorithms via the .rvi configuration file.

**Key differentiator for HydroCraft**: Run 8 model structures on the **same basin, same forcing, same period** and compare performance. This quantifies **structural uncertainty** — how much results vary by model choice. No other HydroCraft model can do this.

### Models Raven Can Emulate (Level 1 — near-exact)

| Emulation | Params | Snow | Climate Suitability |
|-----------|--------|------|---------------------|
| **GR4J** | 4 | No | Humid, semi-humid, tropical |
| **HBV-EC** | 21 | Yes | All climates (default recommendation) |
| **HMETS** | 21 | Yes | Cold, alpine, continental |
| **MOHYSE** | 10 | Yes | Humid, semi-humid |
| **SAC-SMA** | 16 | No | Humid, semi-arid, continental |
| **HYMOD** | 5 | No | Humid, semi-humid |
| **UBC** | 20 | Yes | Cold, alpine, mountainous |
| **HYPR** | 9 | Yes | Cold, prairies |

---

## Installation

### Binary

```
Raven v4.1 executable: KISSPATH_BINARIES/raven/Raven.exe
Source code:            KISSPATH_BINARIES/raven/*.cpp (118 files)
Makefile:               KISSPATH_BINARIES/raven/Makefile
User manual (PDF):      KISSPATH_BINARIES/raven/RavenUsersManual.pdf
```

### Compilation

```bash
cd KISSPATH_BINARIES/raven
g++ -std=c++11 -Wno-deprecated -fPIC -O2 -o Raven.exe *.cpp
chmod +x Raven.exe
```

No external dependencies needed for basic mode. For NetCDF I/O support, uncomment `-Dnetcdf` in Makefile and link libnetcdf.

### Python dependencies

```
numpy, pandas, geopandas, rasterio, shapely, netCDF4 (all in HydroCraft venv)
```

---

## Pipeline (11 stages)

| # | Stage | Tool | Description |
|---|-------|------|-------------|
| s0 | Configuration | `select_model_template.py` | Select emulation template, generate .rvi skeleton |
| s1 | Basin/HRU Setup | `build_rvh_from_shapefile.py` | Generate .rvh from shapefile + DEM + land cover |
| s2 | Parameters | `build_rvp_parameters.py` | Generate .rvp with soil/veg/land use parameters |
| s3 | Forcing | `convert_forcing_to_rvt.py` | Build .rvt straight from CMFD/MSWX/NASA POWER: one point or basin mean (UNIT-CRITICAL) |
| s4 | Model Structure | (via select_model_template.py) | Process algorithm selection in .rvi |
| s5 | Initial Conditions | `generate_rvc_initial.py` | Generate .rvc for cold start |
| s6 | Execution | `run_raven.py` | Run Raven.exe with preflight checks |
| s7 | Output Analysis | `parse_raven_output.py` | Parse Hydrographs.csv, Diagnostics.csv |
| s8 | Multi-Model Ensemble | `run_ensemble_comparison.py` | Run N templates, compare, rank |
| s9 | Calibration | `calibrate_raven_dds.py` | DDS parameter optimization |
| s10 | Coupling | `raven_vic_comparison.py` | Compare Raven vs VIC output |
| -- | Validation | `validate_raven_inputs.py` | Cross-file consistency checks |

**Total**: the current tool count is in the KI map at the top of this file.

### Parallelism

- s1, s2, s3 can run in parallel after s0
- s5 depends on s1 and s2
- s8 (ensemble): all model runs execute in parallel (same forcing/HRU, different .rvi/.rvp)

---

## 6. Output Description

This section restates `dag.yaml`; if this section and `dag.yaml` ever disagree,
`dag.yaml` wins. The headline output is the dag's `validation_rank: 1` variable,
which is the quantity this KI is judged by when observations are supplied.

**Headline output**:

> `discharge (STREAMFLOW)` -- Simulated streamflow hydrograph at each gauged SubBasin outlet; observed series co-reported when supplied. (`m3/s`)

| Output variable (dag `var`) | Validation rank | Emitted in | Unit | Description from `dag.yaml` |
|-----------------------------|----------------:|------------|------|-----------------------------|
| `discharge (STREAMFLOW)` | 1 | `Hydrographs.csv` | `m3/s` | Simulated streamflow hydrograph at each gauged SubBasin outlet; observed series co-reported when supplied. |
| `Actual evapotranspiration (AET)` | 2 | `WatershedStorage.csv` / diagnostics | `mm/d` | Actual ET flux to the atmosphere; diagnostic accumulator, not an independent mass-balance reservoir. |
| `Snow water equivalent (SNOW / TOTAL_SWE)` | 3 | `WatershedStorage.csv` / state outputs | `mm` | Snowpack water equivalent per HRU/basin. |
| `Soil water storage (SOIL[i])` | 4 | `WatershedStorage.csv` | `mm` | Basin-averaged soil layer water storage; SOIL[0]/SOIL[1] can supply soil moisture to coupled crop/water-quality models. |
| `Performance diagnostics (NSE, KGE, PBIAS, RMSE)` | 5 | `Diagnostics.csv` | `dimensionless / m3/s` | Built-in objective functions computed against observed in-stream water discharge (STREAMFLOW) over the simulation period. |

When asked what Raven predicts, answer first with `discharge (STREAMFLOW)`.
Use `parse_raven_output.py` for `Hydrographs.csv` because the raw file is
period-ending and must be shifted to calendar dates before external metrics are
computed.

---

## 8. Unit Conversion Table

Exact input/output shapes live in `docs/format_spec.yaml`. Raven performs no
unit conversion on `.rvt` values, so these conversions must happen before model
execution. They are split in two: the shared loader (`ki_tools_common.load_forcing`)
turns each source into one standard daily series; `tools/s3_forcing/convert_forcing_to_rvt.py`
turns that series into Raven units. Do not redo a loader conversion.

| Variable | Source unit | Raven unit | Conversion | Done by |
|----------|-------------|------------|------------|---------|
| `PRECIP` from CMFD 3-hr | `kg/m2/s` (a rate) | `mm/d` | daily mean rate x `86400` | loader |
| `PRECIP` from MSWX 3-hr | `mm/3hr` | `mm/d` | sum 8 steps | loader |
| `TEMP_MIN`, `TEMP_MAX` from CMFD | `K` | `degC` | subtract `273.15`, daily min / max | loader |
| `TEMP_MIN`, `TEMP_MAX` from MSWX | `degC` | `degC` | daily min / max | loader |
| `TEMP_AVE` | — | `degC` | `(TEMP_MAX + TEMP_MIN) / 2` | s3 tool |
| `SW_RADIA` | `W/m2` | `MJ/m2/d` | multiply by `0.0864` | s3 tool |
| `AIR_PRES` | `Pa` | `kPa` | divide by `1000` | s3 tool |
| `WIND_VEL` | `m/s` | `m/s` | none | — |

### 8c. Sign Conventions and Output Units

| Variable | Convention in this model | Impact if wrong |
|----------|--------------------------|-----------------|
| `discharge (STREAMFLOW)` | Absolute streamflow at gauged SubBasin outlets in `m3/s` | Do not treat as basin-depth runoff or per-grid-cell flux. |
| `Actual evapotranspiration (AET)` | Flux to the atmosphere in `mm/d` | Do not treat as an independent mass-balance reservoir. |
| `Soil water storage (SOIL[i])` | Basin-averaged layer storage in `mm` | Convert or scale before comparison to volumetric soil moisture. |
| `Snow water equivalent (SNOW / TOTAL_SWE)` | Snowpack water equivalent in `mm` | Compare only during periods with meaningful snowpack. |
| `Performance diagnostics (NSE, KGE, PBIAS, RMSE)` | Computed against observed in-stream water discharge when supplied | Missing observations produce diagnostic sentinels, not model validation. |

---

## Site-adaptive s0 flags — READ BEFORE RUNNING A MOUNTAIN BASIN

`select_model_template.py` reads the `.rvh` and `.rvt` already staged in
`--output_dir`, so **stage the basin and forcing files BEFORE calling s0**. It
then makes three site-dependent decisions and reports each in its JSON:

| Flag | Default | What it decides |
|------|---------|-----------------|
| `--pet_method` | `auto` | Substitutes `PET_PRIESTLEY_TAYLOR` for the Appendix-F `PET_OUDIN` when the forcing carries SHORTWAVE. Temperature-index PET collapses at high elevation (dt_rav_039). `template` keeps Appendix F verbatim. |
| `--orographic` | `auto` | Emits `:OroTempCorrect`/`:OroPrecipCorrect OROCORR_SIMPLELAPSE` when the `.rvh` has ≥100 m of relief. **Raven's default is OROCORR_NONE, which silently forces every elevation band with the gauge's own temperature — the elevation-band HRU strategy of s1 does nothing without this** (dt_rav_040). |
| (automatic) | — | Emits `:SWRadiationMethod SW_RAD_DATA` when a radiation-based PET is used and the `.rvt` supplies shortwave; otherwise Raven discards the supplied radiation and drives PET with clear-sky values (dt_rav_041). |

With orographic corrections on, `build_rvp_parameters.py` emits `ADIABATIC_LAPSE`
and `PRECIP_LAPSE` automatically (it queries the binary via `:CreateRVPTemplate`)
and `calibrate_raven_dds.py` calibrates them.

`calibrate_raven_dds.py` takes `--seed`, `--cal_start`, `--cal_end` — **always
pass the calibration window**, otherwise the objective is Raven's whole-simulation
Diagnostics.csv and any held-out score you report afterwards is a fitted number.
It reports `skipped_parameters_absent_from_rvp`: check that list, a name that is
absent was never optimised. Each template's OWN melt parameters are calibrated
(UBCWM multipliers, HMETS min/max melt factors, HBV refreeze) — a wider set needs
a bigger `--n_iterations`, not fewer parameters (dt_rav_043).

Select the reported emulation with `parse_raven_output.select_best_member(...,
cal_key="cal")` — ranking on the held-out window makes structure choice a form
of tuning (dt_rav_036).

## Tools Reference

| Tool | Stage | Script Path | Lines | Purpose |
|------|-------|-------------|------:|---------|
| `select_model_template` | s0 | `tools/s0_config/select_model_template.py` | ~330 | Select from 8 templates; generate .rvi with correct processes |
| `build_rvh_from_shapefile` | s1 | `tools/s1_basin_setup/build_rvh_from_shapefile.py` | ~380 | Generate .rvh from shapefile + DEM + AVHRR; 3 HRU strategies |
| `build_rvp_parameters` | s2 | `tools/s2_parameters/build_rvp_parameters.py` | ~320 | Generate .rvp with soil/veg parameters and template defaults |
| `convert_forcing_to_rvt` | s3 | `tools/s3_forcing/convert_forcing_to_rvt.py` | ~690 | Source (CMFD/MSWX/NASA POWER) -> .rvt, one point or basin mean; observation .rvt; refuses gaps |
| `generate_rvc_initial` | s5 | `tools/s5_initial_conditions/generate_rvc_initial.py` | ~120 | Generate .rvc with climate-aware default initial conditions |
| `run_raven` | s6 | `tools/s6_execution/run_raven.py` | ~290 | Execute Raven with preflight, error parsing, output collection |
| `parse_raven_output` | s7 | `tools/s7_output/parse_raven_output.py` | ~310 | Parse Hydrographs.csv + Diagnostics.csv + WatershedStorage.csv |
| `run_ensemble_comparison` | s8 | `tools/s8_ensemble/run_ensemble_comparison.py` | ~360 | Run N templates on same basin; rank by NSE/KGE; compute spread |
| `calibrate_raven_dds` | s9 | `tools/s9_calibration/calibrate_raven_dds.py` | ~340 | Native Python DDS optimization; modify .rvp and iterate |
| `raven_vic_comparison` | s10 | `tools/s10_coupling/raven_vic_comparison.py` | ~220 | Compare Raven vs VIC discharge; NSE, KGE, correlation |
| `validate_raven_inputs` | all | `tools/common/validate_raven_inputs.py` | ~290 | Cross-file validation: class names, forcing vars, time ranges |

---

## Skill Documents

| Stage | Document | Covers |
|-------|----------|--------|
| s0 | `docs/s0_model_selection_skill.md` | Template selection by climate/data/basin type |
| s1 | `docs/s1_basin_hru_setup_skill.md` | HRU strategies: lumped, elevation bands, land use |
| s3 | `docs/s3_forcing_conversion_skill.md` | Unit conversion table, forcing requirements by PET method |
| s4 | `docs/s4_process_algorithm_guide.md` | Algorithm library reference (120+ algorithms) |
| s8 | `docs/s8_model_intercomparison_skill.md` | Ensemble methodology, structural uncertainty |
| s9 | `docs/s9_calibration_skill.md` | DDS calibration strategy, parameter ranges by template |
| s10 | `docs/coupling_skill.md` | Raven-VIC comparison, Raven-CaMa coupling |

---

## Critical Domain Knowledge

These non-obvious facts cause silent failures. Each is a diagnostic triplet.

### 1. RAVEN IGNORES UNITS (dt_001 through dt_004) — THE #1 TRAP

**"Raven ignores units and will not do units conversion"** — stated explicitly on the Raven cheat sheet. This means:

| Variable | Source Unit | Raven Expects | Conversion | If Wrong |
|----------|---------------|---------------|------------|----------|
| PRECIP | CMFD kg/m2/s (rate); MSWX mm/3hr | mm/d | day total | near-zero or 8x runoff |
| TEMP | CMFD K; MSWX degC | degC | subtract 273.15 (CMFD) | absurd PET, no snowmelt |
| SW_RADIA | W/m2 | MJ/m2/d | multiply 0.0864 | extreme PET |
| AIR_PRES | Pa | kPa | divide 1000 | broken vapor pressure |

**All conversions happen in `ki_tools_common.load_forcing` + `convert_forcing_to_rvt.py`** — never in a hand-made reader (dt_rav_047). The tool refuses values outside the physical range and writes nothing.

### 2. Missing forcing filled with zeros silently (dt_010)

If PRECIP is not in the .rvt file, Raven fills it with zeros. Zero precip = zero discharge. No warning. Always verify PRECIP is present.

### 3. Snow template for snow basins (dt_012)

GR4J, HYMOD, and basic SAC-SMA have NO snow processes. In cold basins, all winter precipitation becomes rain, producing completely wrong seasonal patterns. Use HBV-EC, HMETS, or UBC for snow basins.

### 4. Class names must match exactly between .rvh and .rvp (dt_007, dt_008)

Soil profile names, land use classes, vegetation classes, and terrain classes in the .rvh HRU table must be defined in .rvp. Case-sensitive. Always generate .rvp AFTER .rvh using `--rvh_file` flag.

### 5. Forcing generators produce silent approximations (dt_025)

Raven has 40+ internal forcing generators that estimate missing variables. If WIND_VEL is not provided, Raven estimates it from latitude. These approximations are reasonable but can differ significantly from measurements. Check ForcingFunctions.csv output to see what was generated internally.

### 6. Cold start spinup (dt_023)

Discard first 1-2 years of simulation. Default initial conditions need equilibration time. Or use `solution.rvc` from a previous run for warm start.

### 7. Diagnostics require observed data (dt_011)

If no observed discharge is provided in .rvt, Diagnostics.csv reports -9999 for all metrics. This is not an error — it means "not computed."

### 8. .rvh HRU table requires EXACTLY 13 columns (err_001 — discovered during validation)

The HRU data lines in .rvh must have **13 comma-separated fields**, not the 12 shown in the dissection plan. The undocumented 10th column is **AQUIFER_PROFILE** (set to `[NONE]` if not using aquifer model). Without it, Raven segfaults with "line N is wrong length."

Correct column order: `ID, AREA, ELEVATION, LATITUDE, LONGITUDE, BASIN_ID, LAND_USE_CLASS, VEG_CLASS, SOIL_PROFILE, AQUIFER_PROFILE, TERRAIN_CLASS, SLOPE, ASPECT`

**HMETS default RUNOFF_COEFF**: The default 0.3 is suitable for humid basins. For semi-arid basins, reduce to 0.1-0.2. This is a LandUseParameter, not a GlobalParameter.

### 9. :GlobalParameter is a single-line command, NOT a block (err_002)

Wrong: `:GlobalParameter` / `PARAM1 val` / `:EndGlobalParameter`
Correct: `:GlobalParameter PARAM1 val` (one parameter per line, no block delimiters)

### 10. .rvc file is REQUIRED even if empty (err_004)

Raven v4.1 exits with "Cannot find or read .rvc file" if no .rvc exists. Create a minimal file with just a comment line. Use `generate_rvc_initial.py`.

### 11. Hydrographs.csv is PERIOD-ENDING — never label-join it to a gauge (dt_rav_034)

`Hydrographs.csv`, `ForcingFunctions.csv` and the precip rates in
`WatershedStorage.csv` are **period-ending**: the row stamped date `d` holds the
time-averaged value for the timestep *preceding* `d` (RavenUsersManual v4.1,
"Output files"), and row 0 (stamped `:StartDate`) is only the initial condition.
Observation *input* (`:ObservationData`) has been **period-starting** since v2.7,
so a gauge value for calendar day `d` is written at date `d` and Raven prints it
in the row stamped `d+1`. Consequence: joining a calendar-dated gauge series on
Raven's raw date label scores `sim(day d)` against `obs(day d-1)` — a silent
one-timestep lag in every external metric, and it will not match Raven's own
`Diagnostics.csv`.

```python
# THE canonical reader — applies the period-ending -> calendar-day correction.
sys.path.insert(0, f"{KI}/tools/s7_output")
from parse_raven_output import load_discharge_series, compute_water_balance
sim, raven_obs = load_discharge_series(f"{run_dir}/output", basin_name)   # calendar dates
```

Do **not** write your own `read_hydrograph()` helper in a run script; that is how
the lag gets reintroduced. Proof the correction is right: after the shift,
Raven's own `(observed)` column equals the raw gauge file on identical calendar
dates.

### 12. WatershedStorage "Total [mm]" is NOT physical storage (dt_rav_035)

`Total [mm]` bundles the **`Cum. Losses to Atmosphere` accumulator**. Using `delta(Total)` as ΔS
double-counts ET and reports a large closure FAIL on a run whose native
`MB Error [mm]` closes to machine zero. Use the KI tool:

```python
wb = compute_water_balance(f"{run_dir}/output", basin_name, start=..., end=...)
# P/ET/Q from Cum. Inputs / Cum. Losses to Atmosphere / Cum. Outflow;
# dS from the physical stores only. Also returns raven_mb_error_mm —
# if THAT is large, the fault is genuinely in the model/config.
```
CLI: `parse_raven_output.py --water_balance --wb_start ... --wb_end ...`

### 13. Never select the reported emulation on the held-out window (dt_rav_036)

Choosing "best of N emulations" by its validation-period score (or by Raven's
`Diagnostics.csv` NASH_SUTCLIFFE, which spans the *whole* `:ObservationData`
record, held-out years included) makes the reported held-out number a fitted
statistic. Rank on the calibration window and report the held-out score of the
cal-selected member:

```python
from parse_raven_output import select_best_member
best, ranked = select_best_member(emulation_table, cal_key="cal", metric="nse")
```
`run_ensemble_comparison.py` now emits `rank_basis` (and a `rank_warning` when it
had to fall back to the full-period diagnostic). Drive DDS with
`calibrate_raven_dds.py --cal_start/--cal_end` so the objective itself never sees
the held-out years.

---

## Raven Input File Architecture

| Extension | Name | Purpose | Required |
|-----------|------|---------|----------|
| **.rvi** | Model Definition | Process algorithms, timestep, output options | Yes |
| **.rvp** | Parameters | Soil/veg/land-use class parameters | Yes |
| **.rvh** | HRU/Basin | Subbasins, HRUs (area, elevation, classes) | Yes |
| **.rvt** | Time Series | Meteorological forcing, observed data | Yes |
| **.rvc** | Initial Conditions | Starting state variable values | No (recommended) |

All files share a common prefix (basin name). Raven is invoked as:
```bash
./Raven.exe <basin_name> -o output/
```

---

## Quick Start

```bash
# Activate HydroCraft venv
source KISSPATH_PYTHON_ENV/bin/activate

KI=KISSPATH_KI_ROOT/Raven/knowledge_infrastructure

# 1. Select template (s0 reads the .rvh and .rvt already in --output_dir to decide PET and
#    orographic corrections: for a mountain basin run steps 2 and 4 FIRST, then this step)
python $KI/tools/s0_config/select_model_template.py \
    --template hbv_ec --basin_name chaohe \
    --output_dir outputs/chaohe_raven/ \
    --start_date 2000-01-01 --end_date 2010-12-31

# 2. Generate .rvh (basin/HRU definition)
python $KI/tools/s1_basin_setup/build_rvh_from_shapefile.py \
    --basin_shp data/shp/chaohe_zhangjiaofen_shp/chaohe_zhangjiaofen_boundary_shp/chaohe_zhangjiaofen_boundary.shp \
    --dem data/dem/china_dem_90m/china_dem_90m.tif \
    --output_dir outputs/chaohe_raven/ \
    --basin_name chaohe --strategy elevation_bands --n_bands 5

# 3. Generate .rvp (parameters)
python $KI/tools/s2_parameters/build_rvp_parameters.py \
    --template hbv_ec --basin_name chaohe \
    --rvh_file outputs/chaohe_raven/chaohe.rvh \
    --output_dir outputs/chaohe_raven/

# 4. Build forcing .rvt straight from the source (CRITICAL — unit conversions happen here)
#    basin mean over every CMFD cell centre inside the basin polygon; also writes chaohe_obs.rvt
python $KI/tools/s3_forcing/convert_forcing_to_rvt.py \
    --forcing_source cmfd \
    --forcing_dir KISSPATH_DATA/forcing/Data_forcing_03hr_010deg \
    --basin_shp data/shp/chaohe_zhangjiaofen_shp/chaohe_zhangjiaofen_boundary_shp/chaohe_zhangjiaofen_boundary.shp \
    --output_dir outputs/chaohe_raven/ --basin_name chaohe \
    --start_year 2000 --end_year 2010 --include_full_forcing \
    --obs_file <gauge table with date and Q columns> --obs_subbasin_id 1 \
    --obs_start_date 2001-01-01 --obs_end_date 2010-12-31
#    (a basin inside one or two cells: --lat <lat> --lon <lon> instead of --basin_shp)

# 5. Generate .rvc (initial conditions)
python $KI/tools/s5_initial_conditions/generate_rvc_initial.py \
    --output_dir outputs/chaohe_raven/ --basin_name chaohe \
    --n_hrus 5 --climate semi_humid

# 6. Validate all inputs
python $KI/tools/common/validate_raven_inputs.py \
    --run_dir outputs/chaohe_raven/ --basin_name chaohe

# 7. Run Raven
python $KI/tools/s6_execution/run_raven.py \
    --run_dir outputs/chaohe_raven/ --basin_name chaohe

# 8. Parse output
python $KI/tools/s7_output/parse_raven_output.py \
    --output_dir outputs/chaohe_raven/output/ --basin_name chaohe

# 9. Multi-model ensemble (Raven's killer feature)
python $KI/tools/s8_ensemble/run_ensemble_comparison.py \
    --base_dir outputs/chaohe_raven/ --basin_name chaohe \
    --templates gr4j,hbv_ec,hmets,hymod,sac_sma \
    --start_date 2000-01-01 --end_date 2010-12-31

# 10. Compare with VIC
python $KI/tools/s10_coupling/raven_vic_comparison.py \
    --raven_hydro outputs/chaohe_raven/output/Hydrographs.csv \
    --vic_discharge outputs/chaohe_2000_2010_025deg/routing_param/rout_out/ZJF\ \ .day \
    --output_dir outputs/chaohe_comparison/ --basin_name chaohe
```

---

## Diagnostic Triplets

The table lists the first 25. See `diagnostics/triplets.yaml` for all of them, including the s3 forcing
triplets dt_rav_046 to dt_rav_050 (direct forcing routes, removed VIC and hand-made readers, basin mean,
observation .rvt and gauge elevation).

| ID | Severity | Domain | Summary |
|----|----------|--------|---------|
| dt_001 | silent | unit_conversion | Precip not a day total (mm/d) — 8x overestimate |
| dt_002 | silent | unit_conversion | Temperature in Kelvin not Celsius — wrong PET/snow |
| dt_003 | silent | unit_conversion | Shortwave W/m2 not MJ/m2/d — extreme PET |
| dt_004 | silent | unit_conversion | Pressure Pa not kPa — broken vapor pressure |
| dt_005 | silent | unit_conversion | Timestep mismatch between .rvi and .rvt data |
| dt_006 | fatal | parameter_format | HRU area sum != basin area |
| dt_007 | fatal | parameter_format | Soil profile name mismatch .rvh/.rvp |
| dt_008 | fatal | parameter_format | Land use/vegetation class not defined in .rvp |
| dt_009 | fatal | parameter_format | Data count mismatch in .rvt |
| dt_010 | silent | silent_error | Missing forcing filled with zeros — zero discharge |
| dt_011 | silent | silent_error | No observed data — diagnostics report -9999 |
| dt_012 | silent | silent_error | No snow process for snow-dominated basin |
| dt_013 | fatal | dependency_mismatch | Incompatible algorithm combinations |
| dt_014 | degraded | dependency_mismatch | PET method requires forcing not provided |
| dt_015 | degraded | calibration | DDS not converging — wrong template or narrow ranges |
| dt_016 | degraded | calibration | Overfitting — good calibration, poor validation |
| dt_017 | fatal | runtime | Mass balance error — timestep too large |
| dt_018 | fatal | runtime | Segfault — too many HRUs or long period |
| dt_019 | silent | coupling | Raven-VIC disagreement (expected structural uncertainty) |
| dt_020 | silent | coupling | Timing difference Raven vs VIC (different routing) |
| dt_021 | fatal | compilation | NetCDF linker error |
| dt_022 | fatal | runtime | Binary not executable |
| dt_023 | degraded | silent_error | Cold start spinup artifact |
| dt_024 | silent | silent_error | Ensemble members identical (shared .rvp) |
| dt_025 | silent | silent_error | Raven silently generates missing forcing |

---

## 11. Validation Status

Run results (site, period, scores) are not kept here. They live in the run records
(`test_runs` in hydrocraft.db and each run's own result files).

### Performance Metrics -- Convention Bars

These bars restate `docs/validation_convention.yaml`; if this section and the
convention file disagree, `docs/validation_convention.yaml` wins. The headline field bar for
`discharge (STREAMFLOW)` applies to gauged discharge validation after spinup.

| Dag variable | Obs shape | Metric | Direction | Convention bar, cited |
|--------------|-----------|--------|-----------|-----------------------|
| `discharge (STREAMFLOW)` | `point_time_series` | `nse` | maximize | satisfactory >= `0.50` (`moriasi2015`, `moriasi2007`); good >= `0.70` (`moriasi2015`, `moriasi2007`); very good >= `0.80` (`moriasi2015`, `moriasi2007`) |
| `discharge (STREAMFLOW)` | `point_time_series` | `pbias` | zero_centered | satisfactory within `15` percent (`moriasi2015`); good within `10` percent (`moriasi2015`); very good within `5` percent (`moriasi2015`) |
| `Soil water storage (SOIL[i])` | `point_time_series` | `ubrmse` | minimize | satisfactory <= `0.06` (`smap2017`); good <= `0.04` (`smap2017`); very good: no cited threshold (`smap2017`) |
| `Soil water storage (SOIL[i])` | `point_time_series` | `r` | maximize | satisfactory >= `0.70` (`gripgl2022`); good >= `0.80` (`gripgl2022`); very good >= `0.90` (`gripgl2022`) |
| `Soil water storage (SOIL[i])` | `spatial_time_series` | `r` | maximize | satisfactory >= `0.70` (`gripgl2022`); good >= `0.80` (`gripgl2022`); very good >= `0.90` (`gripgl2022`) |

### Errors found in past runs

1. **err_005, err_006**: faults of an early forcing route that read another model's forcing files. That route was removed on 2026-10-03 (dt_rav_046); the forcing is now built straight from the source.
2. **err_007 (MODERATE)**: `build_rvp_parameters.py` HRU parser used `line.split()` on comma-separated .rvh files, producing wrong class names. **Fixed** by detecting and handling CSV format.
3. **err_008 (MODERATE)**: `build_rvp_parameters.py` read column [9] (AQUIFER_PROFILE = `[NONE]`) as terrain class instead of column [10]. **Fixed** by reading correct column index.
4. **err_009 (CRITICAL)**: `select_model_template.py` GR4J template put `GR4J_X1-X4` as `:GlobalParameter` but Raven v4.1 requires them as soil/land-use properties. Template also used `:SWCanopyCorrect` (not recognized in v4.1). Generic tools need refactoring.

---

## Comparison with Other HydroCraft Models

| Feature | VIC 5.1.0 | WRF-Hydro 5.2.0 | Raven 4.1 |
|---------|-----------|------------------|-----------|
| Grid type | Regular lat/lon | Lambert Conformal Conic | Irregular HRUs |
| Model structures | 1 (VIC) | 1 (Noah-MP) | **8+ emulations** |
| Process algorithms | ~10 | ~5 per category | **120+** |
| Timestep | Sub-daily | Hourly | Daily (configurable) |
| Routing | External | Integrated | Optional |
| Calibration params | 6 (binfilt, Ds...) | ~24 | 4-21 (varies by template) |
| Setup complexity | Grid + soil + veg | 8 domain files | 4 text files (.rv*) |
| Built-in diagnostics | No | No | **18+ metrics** |
| Ensemble capability | No | No | **Yes (unique)** |

---

## File Structure

```
knowledge_infrastructure/
  SKILL.md                                    # This file (agent entry point)
  DISSECTION_PLAN.md                         # Planning document
  knowledge_infrastructure.yaml              # Schema-compliant package definition
  workflow/
    workflow.md                              # Pipeline workflow document
  tools/
    s0_config/select_model_template.py       # Select emulation template
    s1_basin_setup/build_rvh_from_shapefile.py  # Generate .rvh
    s2_parameters/build_rvp_parameters.py    # Generate .rvp
    s3_forcing/convert_forcing_to_rvt.py     # Source -> .rvt, point or basin mean (UNIT-CRITICAL)
    s5_initial_conditions/generate_rvc_initial.py  # Generate .rvc
    s6_execution/run_raven.py                # Execution wrapper
    s7_output/parse_raven_output.py          # Parse output files
    s8_ensemble/run_ensemble_comparison.py   # Multi-model ensemble
    s9_calibration/calibrate_raven_dds.py    # DDS calibration
    s10_coupling/raven_vic_comparison.py     # Raven vs VIC comparison
    common/validate_raven_inputs.py          # Cross-file validation
  docs/
    s0_model_selection_skill.md
    s1_basin_hru_setup_skill.md
    s3_forcing_conversion_skill.md
    s4_process_algorithm_guide.md
    s8_model_intercomparison_skill.md
    s9_calibration_skill.md
    coupling_skill.md
  diagnostics/
    triplets.yaml                           # diagnostic triplets (count: KI map)
    error_log.yaml                          # Error log from real runs
```