Wild Tab Eval

---
name: wild-tab-eval
description: Evaluates the out-of-distribution (OOD) generalization capability of tabular regression models by measuring performance gaps between in-distribution and out-of-distribution test sets. It probes whether advanced OOD training strategies or complex architectures can reliably outperform simple Empirical Risk Minimization (ERM) on unseen data distributions. Use when the user wants to benchmark on VPower_S, VPower_R, Weather, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.01792
  bibtex_key: kolesnikov2023wildtab
  confidence: high
---

# wild-tab-eval

> Wild-Tab: A Benchmark For Out-Of-Distribution Generalization In Tabular Regression — Kolesnikov (2023) (arXiv:2312.01792, 2023)

## What this evaluates

Evaluates the out-of-distribution (OOD) generalization capability of tabular regression models by measuring performance gaps between in-distribution and out-of-distribution test sets. It probes whether advanced OOD training strategies or complex architectures can reliably outperform simple Empirical Risk Minimization (ERM) on unseen data distributions.

## Datasets

- **VPower_S** — total ?; splits: train (-1), val (-1), test (-1)
- **VPower_R** — total ?; splits: train (-1), val (-1), test (-1)
- **Weather** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `MAE` **(primary)** — range: other (kW or Celsius)
  - Mean Absolute Error: the average of the absolute differences between predicted and true values. Formula: MAE = (1/N) * Σ|y_true - y_pred|.
- `RMSE` — range: other (kW or Celsius)
  - Root Mean Squared Error: the square root of the average of squared differences between predicted and true values.
- `MAPE` — range: percent
  - Mean Absolute Percentage Error: the average of absolute percentage errors between predicted and true values.

## Input / output format

**Input**: Tabular feature vectors containing numerical and vector-based features representing real-world industrial data (e.g., weather conditions, power consumption metrics).

**Output**: Continuous scalar regression target (e.g., power consumption in kW or temperature in Celsius).

## Scoring recipe

```python
def compute_mae(y_true, y_pred):
    import numpy as np
    return np.mean(np.abs(np.array(y_true) - np.array(y_pred)))
```

## Common pitfalls

- Validation performance on OOD data does not reliably forecast OOD test performance; strong validation results do not ensure robust OOD generalization.
- Advanced OOD methods (e.g., IRM, IB-ERM) are highly sensitive to hyperparameter tuning and often fail to outperform simple Empirical Risk Minimization (ERM) without careful configuration.
- Using average-in-domain validation for model selection can mislead performance rankings compared to average-out-domain validation, which is better suited for OOD tasks.

## Evidence (verbatim from paper)

> Each dataset’s performance is measured via the MAE, expressed in kW for $VPower_{S}$ and $VPower_{R}$ datasets and in Celsius for the $Weather$ dataset. Supplementary results using RMSE and MAPE metrics are available in [Appendix C](#A3 "Appendix C Additional Experimental Results ‣ Wild-Tab: A Benchmark For Out-Of-Distribution Generalization In Tabular Regression"). In all scenarios, smaller figures denote superior outcomes.

## Citation

```bibtex
@misc{kolesnikov2023wildtab,
  title={Wild-Tab: A Benchmark For Out-Of-Distribution Generalization In Tabular Regression},
  author={Kolesnikov (2023)},
  year={2023},
  note={arXiv:2312.01792}
}
```

- arXiv: 2312.01792