Data Similarity Performance Eval

---
name: data-similarity-performance-eval
description: Evaluates whether distributional or embedding similarity between a model's pretraining data and downstream tasks predicts few-shot or finetuned performance. It probes the 'similarity hypothesis' by measuring correlations between aggregate and example-level text similarities and model accuracy. Use when the user wants to benchmark on BIG-bench Lite, GLUE, or asks about evaluating this task. Reports correlation coefficient.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.09006
  bibtex_key: yauney2023data
  confidence: high
---

# data-similarity-performance-eval

> Data Similarity is Not Enough to Explain Language Model Performance — Yauney et al. (2023) (arXiv:2311.09006, 2023)

## What this evaluates

Evaluates whether distributional or embedding similarity between a model's pretraining data and downstream tasks predicts few-shot or finetuned performance. It probes the 'similarity hypothesis' by measuring correlations between aggregate and example-level text similarities and model accuracy.

## Datasets

- **BIG-bench Lite** — total ?; splits: test (-1)
- **GLUE** — total ?; splits: val (-1), train (-1)

## Metrics

- `correlation coefficient` **(primary)** — range: other
  - Pearson or Spearman correlation between aggregate/example-level similarity scores and few-shot or finetuned accuracy. Tested for statistical significance with Bonferroni correction.
- `accuracy` — range: [0, 1]
  - Fraction of correctly classified examples in few-shot or finetuned settings.
- `embedding cosine similarity` — range: other
  - Cosine similarity between example embeddings and pretraining document embeddings.
- `KL-divergence` — range: other
  - Kullback-Leibler divergence between token distributions of the task and pretraining data.
- `MAUVE` — range: [0, 1]
  - Metric for measuring distributional similarity between text generations or datasets based on human-like vs machine-like text distributions.

## Input / output format

**Input**: Downstream task examples (from BIG-bench Lite or GLUE validation) paired with pretraining corpus documents (from The Pile or C4) for similarity computation; few-shot prompts or finetuning data for model inference.

**Output**: Model predictions (class labels or scores) for each example, used to compute accuracy and correlate with similarity scores.

## Scoring recipe

```python
def evaluate(predictions, gold, sim_scores, num_metrics):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    corr, p_val = pearsonr(sim_scores, gold)
    adj_p = p_val * num_metrics
    return {
        'accuracy': acc,
        'correlation': corr,
        'significant': adj_p < 0.0017
    }
```

## Common pitfalls

- Failing to apply Bonferroni correction when testing multiple similarity metrics against performance, leading to false positives.
- Confusing aggregate dataset similarity with example-level similarity, which yield different conclusions about task difficulty.
- Assuming high similarity guarantees correct classification, as the study shows no consistent example-level correlation.

## Evidence (verbatim from paper)

> Table 1 shows the lack of correlation between five aggregate similarity measures and few-shot performance on BIG-bench Lite tasks. No correlation is statistically significant at the level of p < 0.0017 after Bonferroni correction for multiple tests (Dror et al., 2017). The strongest correlations are for KL-divergence and MAUVE with performance of Flan-T5 XL, the best performing model.

## Citation

```bibtex
@misc{yauney2023data,
  title={Data Similarity is Not Enough to Explain Language Model Performance},
  author={Yauney et al. (2023)},
  year={2023},
  note={arXiv:2311.09006}
}
```

- arXiv: 2311.09006