Bmdal Regression Eval

---
name: bmdal-regression-eval
description: Evaluates the sample efficiency and predictive accuracy of batch-mode deep active learning methods for tabular regression tasks. It probes how well different kernel-based selection strategies reduce prediction error over sequential labeling rounds compared to random sampling. Use when the user wants to benchmark on UCI & OpenML Tabular Regression Benchmark, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.09410
  bibtex_key: holzmuller2022bmdal
  confidence: high
---

# bmdal-regression-eval

> A Framework and Benchmark for Deep Batch Active Learning for Regression — Holzmüller et al. (2022) (arXiv:2203.09410, 2022)

## What this evaluates

Evaluates the sample efficiency and predictive accuracy of batch-mode deep active learning methods for tabular regression tasks. It probes how well different kernel-based selection strategies reduce prediction error over sequential labeling rounds compared to random sampling.

## Datasets

- **UCI & OpenML Tabular Regression Benchmark** — total ?; splits: train (-1), val (-1), pool (-1), test (-1); repo https://github.com/dholzmueller/bmdal_reg

## Metrics

- `RMSE` **(primary)** — range: other
  - Root Mean Squared Error on the test set. The paper reports the mean of the logarithms of RMSE over 20 random seeds, 16 active learning steps, and 15 datasets. A difference of δ between two log values corresponds to a ratio e^δ ≈ 1+δ between the geometric mean errors.

## Input / output format

**Input**: Tabular feature vectors (2 to 379 dimensions) from UCI/OpenML repositories.

**Output**: Continuous regression target values predicted by a 3-layer fully connected neural network.

## Scoring recipe

```python
errors = []
for rep in range(20):
    for step in range(16):
        preds = model.predict(test_X)
        rmse = np.sqrt(np.mean((preds - test_y)**2))
        errors.append(np.log(rmse))
return np.mean(errors)
```

## Common pitfalls

- Averaging the logarithms of error metrics rather than taking the log of the average error, which changes the interpretation to geometric mean reduction.
- Comparing methods that require ensembles or dropout, as they alter the base neural network's error landscape and are not directly comparable to single-model baselines.
- Ignoring batch size sensitivity, as naive methods like MAXDIAG degrade significantly when batch size exceeds ~256.

## Evidence (verbatim from paper)

> Our most important metric is the RMSE, but we will also put some focus on MAXE since it can be interpreted as a measure of robustness to distribution shifts. Generally, RMSE is more affected by rare but large errors than MAE, while the quantiles and MAXE exclusively focus on rare but large errors.

## Citation

```bibtex
@misc{holzmuller2022bmdal,
  title={A Framework and Benchmark for Deep Batch Active Learning for Regression},
  author={Holzmüller et al. (2022)},
  year={2022},
  note={arXiv:2203.09410}
}
```

- arXiv: 2203.09410