Relative Decrease In Mae

---
name: relative-decrease-in-mae
description: Evaluates the effectiveness of machine-learned preconditioners for linear solvers in semi-implicit shallow-water models by measuring error reduction in the first solver iteration and overall convergence rates during free-running simulations. Use when the user has predictions and gold and needs to compute Relative decrease in MAE.
metadata:
  skill_kind: metric
  source_arxiv: 2010.02866
  bibtex_key: ackmann2020mlprecon
  confidence: high
---

# relative-decrease-in-mae

> Machine-Learned Preconditioners for Linear Solvers in Geophysical Fluid Flows — Ackmann et al. (2020) (arXiv:2010.02866, 2020)

## What this evaluates

Evaluates the effectiveness of machine-learned preconditioners for linear solvers in semi-implicit shallow-water models by measuring error reduction in the first solver iteration and overall convergence rates during free-running simulations.

## Datasets

- **Shallow-Water Model Simulation** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/JanAckmann/MLPrecon

## Metrics

- `Relative decrease in MAE` **(primary)** — range: other
  - Calculated as the reduction in Mean Absolute Error at each latitude for the first solver iteration, comparing the simulation using the implicit Richardson preconditioner to the machine-learned preconditioner. Lower values indicate better performance.
- `Convergence rate` — range: other
  - Measured by tracking the normalized residual norms $\| r_{\nu} \|_{\infty} / (\epsilon \| r_0 \|_{\infty})$ across solver iterations during free-running simulations.

## Input / output format

**Input**: Local stencil of residuals $r_0$ (e.g., 5x5 grid or optimized 5-point coordinates) from the shallow-water model state.

**Output**: Predicted correction term $\Delta \tilde{\Phi}$ (or linear regression coefficients $A^{ij}, B^i$).

## Scoring recipe

```python
def compute_relative_mae_decrease(mae_richardson, mae_ml):
    return mae_ml / mae_richardson  # Lower is better

def compute_convergence_rate(residual_nu, residual_0, epsilon):
    norm_nu = np.linalg.norm(residual_nu, ord=np.inf)
    norm_0 = np.linalg.norm(residual_0, ord=np.inf)
    return norm_nu / (epsilon * norm_0)

# Track convergence_rate over solver iterations for free-running simulation
```

## Common pitfalls

- Preconditioners may actually increase error for very small $\Delta \Phi$ values due to incomplete information.
- Training data only covers the first solver iteration, but evaluation tests all iterations in free-running simulations.
- Computational savings depend heavily on balancing iteration count against per-application FLOPs.

## Evidence (verbatim from paper)

> To get a first impression about the quality of results, we compare the relative decrease in Mean Absolute Error (MAE) at each latitude for the first iteration of the solver between the simulation using the implicit Richardson and the Neural Network preconditioner in Figure 2(a). Lower values mean better performance, i.e. higher error reduction. The shown maxima, minima, and median values of the residual norms $\| r_{\nu} \|_{\infty}$ are normalized by $\epsilon \| r_0 \|_{\infty}$.

## Citation

```bibtex
@misc{ackmann2020mlprecon,
  title={Machine-Learned Preconditioners for Linear Solvers in Geophysical Fluid Flows},
  author={Ackmann et al. (2020)},
  year={2020},
  note={arXiv:2010.02866}
}
```

- arXiv: 2010.02866