Tabshap Eval

---
name: tabshap-eval
description: This protocol evaluates the faithfulness of feature attributions for LLM-based tabular classifiers. It measures how well an attribution method ranks features by sequentially masking them in importance order and tracking the resulting drop in the model's predicted class probability. Use when the user wants to benchmark on Adult Income, Heart Disease, or asks about evaluating this task. Reports faithfulness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21120
  bibtex_key: chaudhary2026tabshap
  confidence: high
---

# tabshap-eval

> TabSHAP — Chaudhary et al. (2026) (arXiv:2604.21120, 2026)

## What this evaluates

This protocol evaluates the faithfulness of feature attributions for LLM-based tabular classifiers. It measures how well an attribution method ranks features by sequentially masking them in importance order and tracking the resulting drop in the model's predicted class probability.

## Datasets

- **Adult Income** — total 48842; splits: test (-1)
- **Heart Disease** — total 1025; splits: test (-1)

## Metrics

- `faithfulness` **(primary)** — range: [0, 1]
  - Mean probability mass on the originally predicted class (aggregated over top-K logits) as features are sequentially deleted in order of attribution importance. Plotted against the fraction of features removed, normalized by the average number of features per instance.
- `spearman_rank_correlation` — range: [-1, 1]
  - Spearman's rank correlation coefficient (ρ) measuring the monotonic relationship between feature importance rankings produced by TabSHAP and a baseline (e.g., XGBoost TreeSHAP).

## Input / output format

**Input**: Serialized tabular instance formatted as atomic key:value feature pairs within a ### Input: block, accompanied by a fixed instruction and response template.

**Output**: Class probability distribution (specifically, top-10 logits aggregated to compute class-level probabilities).

## Scoring recipe

```python
def compute_faithfulness(model, instances, attributions):
    scores = []
    for inst in instances:
        orig_dist = model.get_class_distribution(inst)
        orig_class_prob = orig_dist[inst.prediction]
        sorted_feats = sort_features_by_importance(inst.features, attributions[inst.id])
        for k in range(1, len(sorted_feats) + 1):
            masked_inst = remove_features(inst, sorted_feats[:k])
            masked_dist = model.get_class_distribution(masked_inst)
            masked_prob = masked_dist[inst.prediction]
            scores.append(masked_prob)
    return mean(scores) # plotted vs fraction of features removed
```

## Common pitfalls

- Masking subword tokens instead of atomic key-value feature pairs, which corrupts prompt semantics and invalidates the attribution.
- Failing to normalize the deletion fraction by the average number of features per instance, making cross-dataset curve comparisons invalid.
- Using stochastic decoding during probing, which introduces sampling noise that confounds the faithfulness signal with generation variance.

## Evidence (verbatim from paper)

> We evaluate TabSHAP on two distinct tabular tasks to demonstrate its faithfulness, alignment with established baselines, and ability to capture logical constraints in multiclass settings.

## Citation

```bibtex
@misc{chaudhary2026tabshap,
  title={TabSHAP},
  author={Chaudhary et al. (2026)},
  year={2026},
  note={arXiv:2604.21120}
}
```

- arXiv: 2604.21120