Tabular Attention Vs Contrastive Eval

---
name: tabular-attention-vs-contrastive-eval
description: Evaluates the performance of traditional machine learning, deep learning, attention-based, and contrastive learning methods on tabular classification tasks. It probes how data characteristics (dimensionality, difficulty) influence the optimal learning strategy and compares different masking/filling strategies used in contrastive learning. Use when the user wants to benchmark on OpenML Tabular Benchmark, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.04266
  bibtex_key: rabbani2024attention
  confidence: high
---

# tabular-attention-vs-contrastive-eval

> Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking — Rabbani et al. (2024) (arXiv:2401.04266, 2024)

## What this evaluates

Evaluates the performance of traditional machine learning, deep learning, attention-based, and contrastive learning methods on tabular classification tasks. It probes how data characteristics (dimensionality, difficulty) influence the optimal learning strategy and compares different masking/filling strategies used in contrastive learning.

## Datasets

- **OpenML Tabular Benchmark** — total ?; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for classification. Reported as mean scores with standard deviations in parentheses across multiple runs.
- `Win ratio` — range: [0, 28]
  - Count of datasets where method A statistically outperforms method B, determined via Wilcoxon-signed rank test.

## Input / output format

**Input**: Tabular feature matrix containing numerical and categorical columns with corresponding class labels.

**Output**: Predicted class labels or class probabilities for each instance.

## Scoring recipe

```python
# For each of the 28 datasets:
# 1. Train each model (LR, GBT, DNN, DNN-AE, TabNet, FTT, NPT, Pass, Noise, Sample, CutMix, RFC, WCR, SAINT)
# 2. Generate predictions on the test set
# 3. Compute F1 score for each model
# 4. Rank models by F1 score (1 = best)
# 5. Average ranks across all 28 datasets
# 6. Resolve ties using lower standard deviation of F1 scores
# 7. Apply Wilcoxon-signed rank test for pairwise statistical significance
```

## Common pitfalls

- Several deep models (NPT, TabNet) run out of memory on specific datasets, requiring exclusion from rank averages for those cases.
- Rank ties are broken by comparing the standard deviation of F1 scores across runs, not just mean scores.
- Statistical significance is assessed via Wilcoxon-signed rank test, not just raw F1 differences.

## Evidence (verbatim from paper)

> CutMix yields the highest F1 scores on eight out of 28 data sets. Among these eight data sets, hard and easy-to-classify datasets are evenly split. RFC also shows the best F1 scores on eight other data sets, where hard and easy datasets are in a six-to-two ratio. The average rank (AvgRank) in Table 6 across 28 data sets reveals that CutMix (AvgRank: 6.75) and RFC (AvgRank: 6.96) are superior to traditional data corruption methods, including adding noise (Noise, AvgRank: 8.89) and sampling from distribution (Sample, AvgRank: 7.11).

## Citation

```bibtex
@misc{rabbani2024attention,
  title={Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking},
  author={Rabbani et al. (2024)},
  year={2024},
  note={arXiv:2401.04266}
}
```

- arXiv: 2401.04266