Carte Tabular Eval

---
name: carte-tabular-eval
description: Evaluates a graph-based neural architecture for tabular learning on single and multiple tables, testing its ability to handle mixed numerical and categorical features without requiring schema or entity matching. Use when the user wants to benchmark on TabLLM datasets, Entity Matching datasets, or asks about evaluating this task. Reports performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.16785
  bibtex_key: kim2024carte
  confidence: high
---

# carte-tabular-eval

> CARTE: Pretraining and Transfer for Tabular Learning — Kim et al. (2024) (arXiv:2402.16785, 2024)

## What this evaluates

Evaluates a graph-based neural architecture for tabular learning on single and multiple tables, testing its ability to handle mixed numerical and categorical features without requiring schema or entity matching.

## Datasets

- **TabLLM datasets** — total ?; splits: train (-1), test (-1); repo https://github.com/soda-inria/carte
- **Entity Matching datasets** — total ?; splits: train (-1), test (-1); repo https://github.com/soda-inria/carte

## Metrics

- `performance` **(primary)** — range: [0, 1]
  - Standard classification accuracy (fraction of correct predictions) for classification tasks, and R² or RMSE for regression tasks. Reported values in tables are 0–1, indicating accuracy or AUC.

## Input / output format

**Input**: Tabular data containing numerical and categorical features, converted into a graph representation with string embeddings for entries and column names.

**Output**: Predicted class labels or regression values.

## Scoring recipe

```python
def compute_metric(predictions, gold):
    if task == 'classification':
        return (predictions == gold).mean()
    else:
        return r2_score(gold, predictions)
# Results reported as mean ± std over multiple train/test splits or random seeds.
```

## Common pitfalls

- Evaluations use few-shot settings with varying training sizes (32 to 1024 samples), not fixed standard splits.
- Performance is averaged over multiple runs; reporting only a single run will not match the paper's results.
- Schema-matching is explicitly shown to be unnecessary for CARTE's multi-table transfer.

## Evidence (verbatim from paper)

> Table 5 shows the dataset specifications and detailed results on performance comparison between CARTE and baselines presented in Hegselmann et al. (2023). The datasets generally contain high fraction of numerical features (four datasets) or categorical columns with low cardinality (eight datasets).

## Citation

```bibtex
@misc{kim2024carte,
  title={CARTE: Pretraining and Transfer for Tabular Learning},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2402.16785}
}
```

- arXiv: 2402.16785