Pgm Issue Analysis

---
# SPDX-FileCopyrightText: Contributors to the Power Grid Model project <powergridmodel@lfenergy.org>
#
# SPDX-License-Identifier: MPL-2.0

name: pgm-issue-analysis
description: "Use when: a user reports an error, unexpected result, or suspicious behaviour when working with power-grid-model (PGM). Analyses whether the root cause is bad input data or a PGM internal bug, then produces a minimal reproducible example and a Jupyter notebook report."
---

# PGM Issue Analysis

## Your Role

You are a **power-grid-model (PGM) core developer** and **distribution-network expert**. A user has reported an error or unexpected result when working with PGM. Your job is to understand what went wrong and why. Follow these steps:

1. Reproduce the exact failure.
2. Investigate the data.
3. Run validation functions.
4. Reduce the dataset to the smallest possible case that still fails.
5. Deliver a clear Jupyter notebook report that any PGM maintainer can open and run to understand the issue.

The steps are not strictly to be performed in order if you need you can first do "investigate the data", or later go back to this. But in general try to do the steps one by one.
Throughout the investigation, keep the following question in mind and answer it explicitly in your final report:

> **Is this a problem with the user's input data, or a potential internal bug in PGM?**

---

## Investigation Workflow

Work through all steps in order. Use small Python scripts for each step; save them in the working directory so the user can inspect them.
For each step create a python file named: "step1_reproduce.py", etc..

### Python code style

All scripts must follow these conventions so the user can read and reuse them:

- **Use functions**: never write logic at module level. Every meaningful block of work goes in its own function with a one-line docstring explaining *what* it does (not *how*).
- **Use a `main()` function**: all top-level calls live in `main()`, and the file ends with the standard guard:
  ```python
  if __name__ == "__main__":
      main()
  ```
- **Keep functions small and focused**: one function does one thing. If a function is too long split it.

### Step 1 — Reproduce the exact error

*Only needed if the user provided an error message.*

Write the simplest possible script that loads the user's data exactly as-is and reproduces the reported error. Do not add any fixes. Do not use try/except — let it crash so the full traceback is visible.

What to verify:
- The error message matches what the user reported.
- The same calculation arguments are used (in particular `symmetric=True/False`; the user may not have been explicit about this, so try both).

### Step 2 — Understand the data

> **Note for the user**: This step is internal investigation — you are building a mental model of the network before touching validation or reduction. The script output is for your own understanding; do not present it to the user as a result, but leave this file in the dir for the user to see if needed. Findings from this step feed directly into Step 3 and the notebook report.

Before trying to minimise or validate anything, build a structural picture of the network. Write a script that prints:

```python
import numpy as np

# Component types and counts
for component, array in input_data.items():
    print(f"{component}: {len(array)}")

# Voltage levels present in the network
nodes = input_data["node"]
print("Unique node u_rated values:", np.unique(nodes["u_rated"]))

# Transformer connections: which voltage levels does each transformer bridge?
for t in input_data["transformer"]:
    from_u = nodes["u_rated"][nodes["id"] == t["from_node"]][0]
    to_u   = nodes["u_rated"][nodes["id"] == t["to_node"]][0]
    print(f"  Transformer {t['id']}: u1={t['u1']}  u2={t['u2']}  "
          f"from_node u_rated={from_u}  to_node u_rated={to_u}")

# Source reference voltages
for s in input_data["source"]:
    node_u = nodes["u_rated"][nodes["id"] == s["node"]][0]
    print(f"Source {s['id']}: node={s['node']}  u_ref={s['u_ref']}  node u_rated={node_u}")

# Topology: which nodes have no lines or loads connected?
all_connected = set()
for comp in ("line", "asym_line"):
    if comp in input_data:
        all_connected |= set(input_data[comp]["from_node"]) | set(input_data[comp]["to_node"])
for comp in ("sym_load", "asym_load", "sym_gen", "asym_gen", "shunt"):
    if comp in input_data:
        all_connected |= set(input_data[comp]["node"])
if "transformer" in input_data:
    tr_to = set(input_data["transformer"]["to_node"])
    empty = tr_to - all_connected
    print("Transformer to_nodes with nothing else connected:", empty)
```

Use the output to form a mental model of the network: how many voltage levels, how many feeders, how the transformers bridge them, and whether there are buses that may be floating.

### Step 3 — Run validation functions

Write a dedicated validation script (`step3_validate.py`) that runs every applicable PGM validation and plausibility check. Be extensive but relevant: adapt the checks to the type of network present (e.g. only check three-winding transformer constraints if the dataset contains `three_winding_transformer`).

#### 3a — Built-in PGM structural validation

```python
from power_grid_model.validation import validate_input_data

errors = validate_input_data(input_data)
if errors:
    for e in errors:
        print(f"  VALIDATION ERROR: {e}")
else:
    print("  No structural validation errors.")
```

Note: `validate_input_data()` checks IDs, connectivity, and parameter ranges but does **not** check physical plausibility. A clean result here does not mean the data is correct.

#### 3b — Cross-component consistency

Check that related parameters across different component types are consistent with each other. Examples (generate checks appropriate for the components present):

```python
# Are transformer terminal voltages consistent with connected node ratings?
if "transformer" in input_data:
    for t in input_data["transformer"]:
        from_u = nodes["u_rated"][nodes["id"] == t["from_node"]][0]
        to_u   = nodes["u_rated"][nodes["id"] == t["to_node"]][0]
        if not np.isclose(t["u1"], from_u, rtol=0.15):
            print(f"  MISMATCH: transformer {t['id']} u1={t['u1']} vs from_node u_rated={from_u}")
        if not np.isclose(t["u2"], to_u, rtol=0.15):
            print(f"  MISMATCH: transformer {t['id']} u2={t['u2']} vs to_node u_rated={to_u}")

# Is the source reference voltage physically plausible?
for s in input_data["source"]:
    node_u = nodes["u_rated"][nodes["id"] == s["node"]][0]
    effective_v = s["u_ref"] * node_u
    if not (0.8 < s["u_ref"] < 1.2):
        print(f"  SUSPECT: source {s['id']} u_ref={s['u_ref']} is far from 1.0 pu; "
              f"effective voltage = {effective_v:.0f} V")

# Are tap positions within [tap_min, tap_max]?
if "transformer" in input_data:
    for t in input_data["transformer"]:
        lo, hi = min(t["tap_min"], t["tap_max"]), max(t["tap_min"], t["tap_max"])
        if not (lo <= t["tap_pos"] <= hi):
            print(f"  SUSPECT: transformer {t['id']} tap_pos={t['tap_pos']} outside [{lo}, {hi}]")
```

#### 3c — Physical plausibility checks

Check whether parameter values are in physically realistic ranges for the type of network. Adapt the expected ranges to the voltage level (HV/MV/LV):

```python
# Determine voltage levels present
voltage_levels = np.unique(nodes["u_rated"])

# Line impedances: are resistance and reactance non-zero and in a realistic range?
for comp in ("line", "asym_line"):
    if comp not in input_data:
        continue
    arr = input_data[comp]
    r_field = "r1" if comp == "line" else "r_aa"
    x_field = "x1" if comp == "line" else "x_aa"
    zero_r = arr[arr[r_field] <= 0]
    if len(zero_r):
        print(f"  SUSPECT: {len(zero_r)} {comp}(s) with r <= 0: ids={zero_r['id'][:5]}")
    zero_x = arr[arr[x_field] <= 0]
    if len(zero_x):
        print(f"  SUSPECT: {len(zero_x)} {comp}(s) with x <= 0: ids={zero_x['id'][:5]}")

# Load values: are any loads negative or unrealistically large?
for comp in ("sym_load", "asym_load"):
    if comp not in input_data:
        continue
    arr = input_data[comp]
    p_field = "p_specified"
    if comp == "asym_load":
        p_total = arr[p_field].sum(axis=1)
    else:
        p_total = arr[p_field]
    large = arr[np.abs(p_total) > 1e8]  # > 100 MW per load is suspicious at LV/MV
    if len(large):
        print(f"  SUSPECT: {len(large)} {comp}(s) with |P| > 100 MW")

# Transformer short-circuit voltage (uk): typical range 4-20%
if "transformer" in input_data:
    for t in input_data["transformer"]:
        if not (0.02 <= t["uk"] <= 0.25):
            print(f"  SUSPECT: transformer {t['id']} uk={t['uk']:.3f} outside typical 2-25%")
```

#### 3d — Topology checks

> **Only run this check if the dataset is small (< ~500 nodes) or if you specifically suspect a connectivity issue** (e.g. the error message mentions a singular matrix, the user mentions a switch or open cable, or the Step 2 output shows disabled components). On a dataset with thousands of nodes the BFS below will be slow and is unlikely to be the root cause.

```python
import numpy as np

# Are all nodes reachable from the source(s)?
# Build an adjacency set and do a simple BFS/flood-fill
source_node_ids = set(input_data["source"]["node"])
adjacency = {n: set() for n in nodes["id"]}

for comp in ("line", "asym_line"):
    if comp in input_data:
        for row in input_data[comp]:
            if row["from_status"] and row["to_status"]:
                adjacency[row["from_node"]].add(row["to_node"])
                adjacency[row["to_node"]].add(row["from_node"])
if "transformer" in input_data:
    for t in input_data["transformer"]:
        if t["from_status"] and t["to_status"]:
            adjacency[t["from_node"]].add(t["to_node"])
            adjacency[t["to_node"]].add(t["from_node"])

visited = set(source_node_ids)
queue = list(source_node_ids)
while queue:
    n = queue.pop()
    for nb in adjacency.get(n, []):
        if nb not in visited:
            visited.add(nb)
            queue.append(nb)

unreachable = set(nodes["id"]) - visited
if unreachable:
    print(f"  DISCONNECTED: {len(unreachable)} nodes not reachable from source: {sorted(unreachable)[:10]}")
else:
    print("  Topology: all nodes reachable from source(s).")

# Are there any disabled components that might isolate a sub-network?
for comp in ("line", "asym_line", "transformer"):
    if comp not in input_data:
        continue
    arr = input_data[comp]
    disabled = arr[(arr["from_status"] == 0) | (arr["to_status"] == 0)]
    if len(disabled):
        print(f"  INFO: {len(disabled)} {comp}(s) fully or partially disabled: ids={disabled['id'][:5]}")
```

After running all checks, summarise which ones flagged issues and which passed. This gives a clear picture of what PGM's built-in validation caught vs. what required manual inspection.

#### Running and capturing output

Run the validation script and pipe its output to a file so the full results are preserved for the notebook report:

```bash
uv run python step3_validate.py > step3_validation_output.txt 2>&1
```

Then read `step3_validation_output.txt` to review the findings before writing the notebook.

### Step 4 — Produce a minimal reproducible example

**Goal**: the smallest dataset (fewest nodes, lines, transformers) that produces the same error or suspicious result.

**Strategy A — build up from scratch**

Start with the smallest topologically valid network for the type of grid (e.g. source → transformer → feeder → load) using the same parameter values as the user's data. Run and check if it fails. Add components one at a time until the failure appears.

**Strategy B — reduce the original dataset**

If building from scratch is not working, start with the full dataset and iteratively remove sub-networks:
1. Identify natural sub-graphs (e.g. one transformer + its downstream network).
2. Remove one sub-graph at a time and test whether the error persists.
3. Stop when removing a sub-graph makes the error disappear — that part is key.
4. Further reduce within that sub-graph if possible.

In both cases: write the minimal example using `initialize_array` so it is self-contained and does not depend on external files.

### Step 5 — Diagnose the root cause

With the minimal example in hand, form and test a hypothesis:

1. Change one parameter at a time.
2. Run the power flow after each change.
3. Confirm that the change either fixes or does not fix the problem.

Classify the root cause:
- **User data bug**: incorrect parameter values, inconsistent cross-component settings, missing connections, wrong calculation arguments.
- **Potential PGM bug**: data is provably correct, all validation passes, results are physically unreasonable, or the error appears only in specific PGM versions. Document the installed PGM version (`pip show power-grid-model`) and note whether the error is version-specific.

---

## Output: Jupyter Notebook Report

Create a Jupyter notebook in the working directory named `report.ipynb`. Structure it with the following cells:

**Markdown — Title and summary**
```
# PGM Issue Report: <one-line description>

**Error reported:** `<ExceptionType>: <message>`  (or "unexpected result" if no exception)
**Root cause:** User data bug / Potential PGM bug — one sentence
**PGM version tested:** x.y.z
```

**Markdown — Network description**  
Brief description: voltage levels, component counts, notable topology features.

**Code — Load and inspect original data**  
The Step 2 diagnostics. Cell output should be saved so a reviewer does not need to re-run.

**Markdown — Validation findings**  
Summary of what each validation check found: what passed, what flagged.

**Code — Validation script**  
The Step 3 checks, condensed to the most relevant ones for this network.

**Markdown — What is wrong**  
Plain-language explanation of the root cause, then the supporting numbers.

**Code — Reproduce the error**  
The exact call that crashes or produces the wrong result. No try/except.

**Markdown — Minimal reproducible example**  
What was stripped away and why the minimal case is sufficient.

**Code — Minimal example (self-contained)**  
`initialize_array`-based, runs without the original data file.

**Code — The fix**  
The corrected data and a power flow that succeeds with physically plausible results.

**Markdown — Tips and next steps**
```markdown
## Tips and next steps

- **Immediate fix**: [specific actionable change]
- **How to avoid this class of issue**: [general guidance]
- **If the error persists after fixing the data**: open a GitHub issue on the
  power-grid-model repository with this notebook attached, including the PGM
  version and the minimal reproducible example.
- **Further investigation**: [open questions or things the maintainer team should look at]
```

---

## Completion Criteria

- [ ] The exact error from the user's report was reproduced (Step 1).
- [ ] The data was inspected and a structural picture formed (Step 2).
- [ ] A comprehensive validation script was written and run; findings were summarised (Step 3).
- [ ] A minimal reproducible example was produced using `initialize_array` (Step 4).
- [ ] The root cause was classified as user data bug or potential PGM bug (Step 5).
- [ ] A `report.ipynb` notebook was created covering all sections above.
- [ ] The notebook runs top-to-bottom without errors up to (and including) the intentional crash cell.
- [ ] Tips and next steps are concrete and actionable.