Information Sufficiency

---
name: information-sufficiency
description: Evaluates the quality of text embedding models in a task-agnostic manner by estimating information sufficiency via normalizing flows. It predicts how well an embedding model will perform on downstream tasks without requiring task-specific labels or fine-tuning. Use when the user has predictions and gold and needs to compute Spearman's ρ.
metadata:
  skill_kind: metric
  source_arxiv: 2604.17344
  bibtex_key: jiang2026flare
  confidence: high
---

# information-sufficiency

> FLARE: Task-agnostic embedding model evaluation through a normalization process — Jingzhou Jiang et al. (2026) (arXiv:2604.17344, 2026)

## What this evaluates

Evaluates the quality of text embedding models in a task-agnostic manner by estimating information sufficiency via normalizing flows. It predicts how well an embedding model will perform on downstream tasks without requiring task-specific labels or fine-tuning.

## Datasets

- **Apt-eval** — total ?; splits: test (-1)
- **GT-FintechLab** — total ?; splits: test (-1)
- **BhashaBench-Finance** — total ?; splits: test (-1)
- **AIR-Bench-Finance** — total ?; splits: test (-1)
- **LIMIT** — total ?; splits: test (-1)
- **ArXiv-Abstracts 2025** — total ?; splits: test (-1); HF `almanach/arxiv_abstracts_2025`
- **Augmented STS-B** — total ?; splits: test (-1); HF `maiammar/Augmented_stsb_multi_mt`
- **LivNLP-STS** — total ?; splits: test (-1)
- **Philosophical-STS** — total ?; splits: test (-1); HF `johnnyboycurtis/Philosophical-STS-Text-Pairs`
- **Clustered-FunPang Medical** — total ?; splits: test (-1); HF `mukulb/clustered_FUNPANG_dataset_with_groups`
- **Reasoning-Clustering** — total ?; splits: test (-1); HF `Ibisbill/Clustering_deduplicated_reasoning`

## Metrics

- `Spearman's ρ` **(primary)** — range: [-1, 1]
  - Spearman's rank correlation coefficient between the predicted Information Sufficiency (IS) scores and the ground-truth supervised benchmark scores (F1 macro, nDCG@10, Spearman correlation, or V-measure depending on task).

## Input / output format

**Input**: High-dimensional text embeddings (d=384–4096) generated by candidate models on 11 held-out datasets. For retrieval tasks, only passage corpus embeddings are used.

**Output**: A single scalar Information Sufficiency (IS) score per model per dataset, computed as the median of per-pair IS scores derived from normalizing flow log-likelihoods.

## Scoring recipe

```python
# 1. Compute IS score for each model on each dataset
is_scores = []
for model in models:
    for dataset in datasets:
        embeddings = model.encode(dataset)
        is_score = compute_is_score(embeddings) # via normalizing flows
        is_scores.append(is_score)

# 2. Compute ground-truth supervised metric for each model/dataset
gt_scores = []
for model in models:
    for dataset in datasets:
        gt_scores.append(get_supervised_metric(model, dataset)) # F1, nDCG@10, etc.

# 3. Compute ranking alignment
spearman_rho = spearmanr(is_scores, gt_scores).correlation
```

## Common pitfalls

- Using kernel or GMM density estimators in high-dimensional spaces (d ≥ 3584) causes vanishingly sparse estimates and systematic ranking inversions.
- Assuming embedding spaces should be uniform or penalizing anisotropy fails because high-quality semantic spaces are inherently anisotropic.
- Averaging per-pair IS scores is sensitive to heavy-tailed distributions; median aggregation is required to avoid noise from geometric mismatches.

## Evidence (verbatim from paper)

> We assess the reliability of our unsupervised Information Sufficiency (IS) metric by measuring its alignment with ground-truth supervised rankings. Ground-truth performance is established using standard MTEB metrics: F1 macro for classification, nDCG@10 for retrieval, Spearman correlation for STS, and V-measure for clustering. We quantify ranking alignment using Spearman’s rank correlation ($\rho$) and Pearson correlation ($r$) between the predicted IS scores and the supervised metrics.

## Citation

```bibtex
@misc{jiang2026flare,
  title={FLARE: Task-agnostic embedding model evaluation through a normalization process},
  author={Jingzhou Jiang et al. (2026)},
  year={2026},
  note={arXiv:2604.17344}
}
```

- arXiv: 2604.17344