Tabular Cleaning Eval

---
name: tabular-cleaning-eval
description: Evaluates automated tabular data cleaning pipelines by measuring downstream classification accuracy and calibration when processed by a Tabular Foundation Model (TabPFN v2). It probes whether cleaning strategies can effectively align dirty data distributions with the model's learned prior to improve predictive performance. Use when the user wants to benchmark on OpenML CC18 Benchmark Suite (D1–D10), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.25154
  bibtex_key: [REDACTED]
  confidence: high
---

# tabular-cleaning-eval

> Prior-Aligned Data Cleaning for Tabular Foundation Models — Berti-Equille et al. (2026) (arXiv:2604.25154, 2026)

## What this evaluates

Evaluates automated tabular data cleaning pipelines by measuring downstream classification accuracy and calibration when processed by a Tabular Foundation Model (TabPFN v2). It probes whether cleaning strategies can effectively align dirty data distributions with the model's learned prior to improve predictive performance.

## Datasets

- **OpenML CC18 Benchmark Suite (D1–D10)** — total ?; splits: train (-1), test (-1); repo https://github.com/LaureBerti/Learn2Clean

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted class labels on the held-out test split.
- `ECE` — range: [0, 1]
  - Expected Calibration Error computed over 10 equal-width confidence bins on the softmax probability of the predicted class. Calculated as the weighted average of the absolute difference between bin accuracy and bin confidence.

## Input / output format

**Input**: Tabular dataset (rows × features) containing natural or synthetically injected errors (MCAR, MAR, outliers, duplicates) and missing values.

**Output**: Softmax class probabilities and predicted class label from TabPFN v2.

## Scoring recipe

```python
correct = sum(pred == gold for pred, gold in zip(predictions, gold_labels))
accuracy = correct / len(gold_labels)

bins = 10
bin_width = 1.0 / bins
ece = 0.0
for i in range(bins):
    low, high = i * bin_width, (i + 1) * bin_width
    mask = (confidence >= low) & (confidence < high)
    if mask.sum() > 0:
        bin_acc = (pred[mask] == gold[mask]).mean()
        bin_conf = confidence[mask].mean()
        ece += (mask.sum() / len(gold)) * abs(bin_acc - bin_conf)
```

## Common pitfalls

- Using TabPFN v1 instead of v2, as v1 lacks the unconditional z-normalization, power transform, and missing-value flags that critically affect sensitivity to upstream data quality.
- Relying solely on accuracy for class-imbalanced datasets (e.g., Blood Transfusion, Adult, Bank Marketing), where it may understate minority-class benefits; ECE should be consulted for calibration insights.
- Assuming high reward function scores correlate with high downstream accuracy, as trivial-collapse rewards (e.g., completeness retention) saturate at ~1.0 while yielding poor predictive performance.

## Evidence (verbatim from paper)

> All cleaning policies are finally evaluated by TabPFN v2 accuracy and ECE on a 20% held-out test split (stratified, seed=42). ECE is computed with 10 equal-width confidence bins on the softmax probability of the predicted class. Accuracy is the primary metric for three reasons: (i) it is the standard reported by TabPFN v2’s own benchmark suite*(Hollmann et al., [2025](#bib.bib15 ""))* and the OpenML repository for these tasks, enabling direct comparison with published baselines; (ii) seven of the ten datasets have near-balanced class distributions, where accuracy and AUROC are empirically tightly correlated; and (iii) since all methods are evaluated under identical conditions, the *ranking* of cleaning strategies is robust to the choice of aggregation metric when the pipeline affects the data distribution uniformly across classes—which prior-alignment cleaning does by construction.

## Citation

```bibtex
@misc{berti2026prioraligned,
  title={Prior-Aligned Data Cleaning for Tabular Foundation Models},
  author={Berti-Equille et al. (2026)},
  year={2026},
  note={arXiv:2604.25154}
}
```

- arXiv: 2604.25154