Instance Attribution Eval

---
name: instance-attribution-eval
description: Evaluates the ability of different instance attribution methods to rank training data instances by their influence on a given test prediction, particularly focusing on identifying problematic training artifacts and comparing gradient-based versus similarity-based approaches. Use when the user wants to benchmark on SST-2, MNLI, HANS, or asks about evaluating this task. Reports Spearman Correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.04128
  bibtex_key: pezeshkpour2021instance
  confidence: high
---

# instance-attribution-eval

> An Empirical Comparison of Instance Attribution Methods for NLP — Pezeshkpour et al. (2021) (arXiv:2104.04128, 2021)

## What this evaluates

Evaluates the ability of different instance attribution methods to rank training data instances by their influence on a given test prediction, particularly focusing on identifying problematic training artifacts and comparing gradient-based versus similarity-based approaches.

## Datasets

- **SST-2** — total ?; splits: test (-1)
- **MNLI** — total ?; splits: train (10000), test (-1)
- **HANS** — total 1000; splits: test (1000)

## Metrics

- `Spearman Correlation` **(primary)** — range: [-1, 1]
  - Measures the rank correlation between the importance scores assigned to training instances by two different attribution methods. Computed as the Pearson correlation coefficient between the rank-transformed scores of the training set.

## Input / output format

**Input**: A target test instance, a trained model (BERT-based), and the training dataset (or a sampled subset).

**Output**: A ranked list or continuous importance score for each training instance relative to the target test instance.

## Scoring recipe

```python
def compute_spearman(scores_a, scores_b):
    rank_a = rankdata(scores_a)
    rank_b = rankdata(scores_b)
    n = len(rank_a)
    d_sq = sum((r1 - r2) ** 2 for r1, r2 in zip(rank_a, rank_b))
    return 1.0 - (6.0 * d_sq) / (n * (n ** 2 - 1))
```

## Common pitfalls

- Computing full Influence Functions for BERT is infeasible due to Hessian storage requirements (~12 PB), so approximations like LiSSa or parameter subsets (Top-5 layers, linear layer) must be used.
- The LiSSa approximation for the Inverse Hessian Vector Product is sensitive to the norm of the approximation, often requiring a large scaling factor for convergence and becoming unstable with additional layers.

## Evidence (verbatim from paper)

> The similarity between influence of training samples for different pairs of attribution methods on the SST and MNLI datasets was measured via Spearman Correlation.

## Citation

```bibtex
@misc{pezeshkpour2021instance,
  title={An Empirical Comparison of Instance Attribution Methods for NLP},
  author={Pezeshkpour et al. (2021)},
  year={2021},
  note={arXiv:2104.04128}
}
```

- arXiv: 2104.04128