This metric quantifies the degree of overlap between different LLM benchmarks by comparing their token-level perplexity signatures derived from in-the-wild pretraining corpora, revealing whether performance correlations stem from shared latent capacity familiarity or benchmark-orthogonal factors like question format. Use when the user has predictions and gold and needs to compute signature_overlap.
Installs into .claude/skills of the current project.
Are you the author of Signature Overlap?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/qhjqhj00-signature-overlap)
---
name: signature_overlap
description: This metric quantifies the degree of overlap between different LLM benchmarks by comparing their token-level perplexity signatures derived from in-the-wild pretraining corpora, revealing whether performance correlations stem from shared latent capacity familiarity or benchmark-orthogonal factors like question format. Use when the user has predictions and gold and needs to compute signature_overlap.
metadata:
skill_kind: metric
source_arxiv: 2509.23488
bibtex_key: [REDACTED]
confidence: high
---
# signature_overlap
> Mapping Overlaps in Benchmarks through Perplexity in the Wild — Wu et al. (2025) (arXiv:2509.23488, 2025)
## What this evaluates
This metric quantifies the degree of overlap between different LLM benchmarks by comparing their token-level perplexity signatures derived from in-the-wild pretraining corpora, revealing whether performance correlations stem from shared latent capacity familiarity or benchmark-orthogonal factors like question format.
## Datasets
- **RedPajama** — total ?; splits: (unstated)
## Metrics
- `signature_overlap` **(primary)** — range: [-1, 1]
- Pairwise Pearson correlation between benchmark signature vectors, where signatures are derived via linear regression on token-level perplexity patterns from in-the-wild corpora.
## Input / output format
**Input**: Token sequences from in-the-wild corpora (RedPajama) used to compute per-benchmark token-level perplexity vectors.
**Output**: A scalar correlation coefficient representing the signature overlap between two benchmarks.
## Scoring recipe
```python
# Compute token-level perplexity for each benchmark's questions on RedPajama
perplexity_vectors = {bench: compute_perplexity(bench_questions, corpus) for bench in benchmarks}
# Extract signature via linear regression on pretraining exposure patterns
signatures = {bench: linear_regression(perplexity_vectors[bench]) for bench in benchmarks}
# Compute pairwise correlation between signature vectors
overlap_matrix = {
(b1, b2): pearson_correlation(signatures[b1], signatures[b2])
for b1, b2 in itertools.combinations(benchmarks, 2)
}
return overlap_matrix
```
## Common pitfalls
- Confusing signature overlap with semantic or performance overlap, as the latter show much weaker discriminative ability and are heavily biased by question format.
- Assuming high overlap indicates true semantic alignment, when it often reflects benchmark-orthogonal factors like shared question formats or training data contamination.
- Ignoring the impact of benchmark category size when calculating cross-category overlaps, which requires equal weighting to avoid skewing results.
## Evidence (verbatim from paper)
> We extract benchmark signatures from the open dataset RedPajama (Weber et al., 2024). ... signature overlap (panel a), on the left, we compare within-category overlap against the average cross-category overlap. ... signature-level analysis reveals statistically insignificant tiny differences based on the Mann–Whitney U test, yielding results around 0. This aligns with intuition, as the signature provides a good approximation of the true overlap and variation.
## Citation
```bibtex
@misc{wu2025mappingoverlaps,
title={Mapping Overlaps in Benchmarks through Perplexity in the Wild},
author={Wu et al. (2025)},
year={2025},
note={arXiv:2509.23488}
}
```
- arXiv: 2509.23488
