Stochastic Ackley Eval

---
name: stochastic-ackley-eval
description: Evaluates the ability of uncertainty-aware deep neural networks to approximate a highly irregular, multi-extremum function and quantify predictive uncertainty. It specifically probes how well the models handle in-distribution versus out-of-distribution parameter regimes. Use when the user wants to benchmark on Stochastic Ackley Function, or asks about evaluating this task. Reports Relative Error (RE).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.13506
  bibtex_key: jacquier2020poduqnn
  confidence: high
---

# stochastic-ackley-eval

> Non-Intrusive Reduced-Order Modeling Using Uncertainty-Aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to Flood Modeling — Jacquier et al. (2020) (arXiv:2005.13506, 2020)

## What this evaluates

Evaluates the ability of uncertainty-aware deep neural networks to approximate a highly irregular, multi-extremum function and quantify predictive uncertainty. It specifically probes how well the models handle in-distribution versus out-of-distribution parameter regimes.

## Datasets

- **Stochastic Ackley Function** — total 600; splits: train (500), test (100); repo https://github.com/pierremtb/POD-UQNN

## Metrics

- `Relative Error (RE)` **(primary)** — range: [0, 1] | percent
  - Normalized difference between predicted and true values: RE = mean(|y_pred - y_true|) / mean(|y_true|).
- `Mean Prediction Interval Width (MPIW)` — range: [0, ∞)
  - Average width of the prediction intervals across the dataset, computed as the mean of (upper bound - lower bound) where bounds are mean ± 2×std.

## Input / output format

**Input**: Normalized non-spatial parameters X (size P=3) and 2D spatial coordinates (x,y) over a 400×400 grid (160,000 DOFs).

**Output**: Predicted mean ū_D^μ and predicted standard deviation ū_D^σ for the function value u.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred_mean, y_pred_std):
    re = np.mean(np.abs(y_pred_mean - y_true)) / np.mean(np.abs(y_true))
    interval_width = 4 * y_pred_std  # ±2 std
    mpiw = np.mean(interval_width)
    return re, mpiw
```

## Common pitfalls

- The dataset is synthetically generated via Latin Hypercube Sampling, not a standard public benchmark.
- Validation split is fixed at 20% of the generated dataset, with manual early stopping rather than automated.
- Out-of-distribution evaluation uses a specific disjoint parameter range Ω_out = [-2,-1] ∪ [1,2].

## Evidence (verbatim from paper)

> The relative error RE defined in (21) is computed at each training epoch for both the training set and the validation set. ... To picture the difference in confidence between in- and out-of-scope predictions quantitatively, we computed MPIW_tst=0.15 and MPIW_out=10.0.

## Citation

```bibtex
@misc{jacquier2020poduqnn,
  title={Non-Intrusive Reduced-Order Modeling Using Uncertainty-Aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to Flood Modeling},
  author={Jacquier et al. (2020)},
  year={2020},
  note={arXiv:2005.13506}
}
```

- arXiv: 2005.13506