Tabarena Eval

---
name: tabarena-eval
description: Evaluates the predictive performance of tabular machine learning models across 51 real-world datasets under standardized, reproducible protocols. It probes how hyperparameter tuning, nested cross-validation, and post-hoc ensembling affect peak performance and efficiency trade-offs. Use when the user wants to benchmark on TabArena, or asks about evaluating this task. Reports predictive performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.16791
  bibtex_key: erickson2025tabarena
  confidence: medium
---

# tabarena-eval

> TabArena: A Living Benchmark for Machine Learning on Tabular Data — Nick Erickson et al. (2025) (arXiv:2506.16791, 2025)

## What this evaluates

Evaluates the predictive performance of tabular machine learning models across 51 real-world datasets under standardized, reproducible protocols. It probes how hyperparameter tuning, nested cross-validation, and post-hoc ensembling affect peak performance and efficiency trade-offs.

## Datasets

- **TabArena** — total 51; splits: train (-1), test (-1)

## Metrics

- `predictive performance` **(primary)** — range: other
  - Task-dependent standard tabular metric: AUC-ROC for classification, R² or RMSE for regression. Aggregated across all datasets.
- `median inference time per 1000 samples` — range: other
  - Wall-clock time to generate predictions on 1000 test samples, reported as median across datasets.

## Input / output format

**Input**: Tabular feature matrix (numerical/categorical) and target column.

**Output**: Predicted target values or class probabilities per instance.

## Scoring recipe

```python
scores = []
for dataset in datasets:
    train, test = split(dataset)
    model = train_with_nested_cv(train)
    preds = model.predict(test)
    scores.append(compute_metric(preds, test.target))
return mean(scores)
```

## Common pitfalls

- Using holdout validation instead of nested cross-validation significantly underestimates model performance and biases rankings toward ensembling methods.
- Evaluating models with default parameters only misrepresents peak performance; post-hoc ensembling is required to reveal true capabilities.
- Ignoring compute and hardware constraints leads to inaccurate assessments of efficiency trade-offs and inference costs.

## Evidence (verbatim from paper)

> Predictive performance of a model with tuning and ensembling when using holdout or cross-validation.

## Citation

```bibtex
@misc{erickson2025tabarena,
  title={TabArena: A Living Benchmark for Machine Learning on Tabular Data},
  author={Nick Erickson et al. (2025)},
  year={2025},
  note={arXiv:2506.16791}
}
```

- arXiv: 2506.16791