Comp Eval

---
name: comp-eval
description: Evaluates a model's ability to align latent representations across different conditions (e.g., batch effects, treatment, demographic attributes) while preserving task-relevant information. It measures local mixing quality using nearest-neighbour and silhouette metrics, and assesses predictive utility via classification accuracy on held-out labels. Use when the user wants to benchmark on Tumour / Cell Line, Stimulated / untreated single-cell PBMCs, Single-cell RNA-seq data integration (PBMCs), UCI Adult Income, or asks about evaluating this task. Reports kBET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.08161
  bibtex_key: foster2021comp
  confidence: high
---

# comp-eval

> Contrastive Mixture of Posteriors for Counterfactual Inference, Data Integration and Fairness — Foster et al. (2021) (arXiv:2106.08161, 2021)

## What this evaluates

Evaluates a model's ability to align latent representations across different conditions (e.g., batch effects, treatment, demographic attributes) while preserving task-relevant information. It measures local mixing quality using nearest-neighbour and silhouette metrics, and assesses predictive utility via classification accuracy on held-out labels.

## Datasets

- **Tumour / Cell Line** — total ?; splits: (unstated)
- **Stimulated / untreated single-cell PBMCs** — total ?; splits: (unstated)
- **Single-cell RNA-seq data integration (PBMCs)** — total ?; splits: (unstated)
- **UCI Adult Income** — total ?; splits: (unstated)

## Metrics

- `kBET` **(primary)** — range: [0, 1]
  - k-Batch Effect Test (kBET) measures the rejection rate of the null hypothesis that the proportion of samples from a given batch in the k-nearest neighbours matches the global batch proportion. A low value close to zero indicates good local mixing.
- `Silhouette Coefficient (s)` — range: other
  - Local Silhouette Coefficient computed over k-nearest neighbours in the latent space. A low value close to zero indicates good local mixing of sample representations.
- `m-kBET` — range: [0, 1]
  - Mean kBET calculated by computing kBET separately for every fixed subpopulation (e.g., cell type) and averaging the results. Penalizes global alignment that mixes different subpopulations.
- `mean Silhouette Coefficient (\tilde{s})` — range: other
  - Mean Silhouette Coefficient calculated by computing the local Silhouette Coefficient separately for every fixed subpopulation and averaging the results.
- `Accuracy` — range: [0, 1]
  - Standard classification accuracy on held-out labels (e.g., cancer type, income, gender) predicted from the learned latent representations.

## Input / output format

**Input**: Gene expression profiles (bulk or single-cell) or tabular census data, paired with condition labels (e.g., tumour/cell-line, stimulated/untreated, library protocol, gender/income).

**Output**: Latent representations (z) and/or reconstructed gene expression profiles. For classification tasks, predicted class labels.

## Scoring recipe

```python
def compute_kBET(latents, batch_labels, k=100, alpha=0.01):
    # Compute k-NN in latent space
    # For each sample, calculate proportion of neighbors from same batch
    # Return rejection rate (low is good)
    pass

def compute_mean_kBET(latents, batch_labels, subpop_labels):
    scores = [compute_kBET(latents[sp], batch_labels[sp]) for sp in unique(subpop_labels)]
    return mean(scores)

def compute_accuracy(predictions, gold_labels):
    return sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
```

## Common pitfalls

- Low values for kBET and Silhouette Coefficient indicate good mixing, which is counterintuitive compared to standard accuracy metrics.
- The mean variants (m-kBET, \tilde{s}) specifically penalize models that achieve global alignment but erroneously cluster different subpopulations together.
- Evaluation metrics are sometimes computed on 2D UMAP projections rather than the full high-dimensional latent space.

## Evidence (verbatim from paper)

> To benchmark CoMP on the first objective, we use the following pair of k nearest-neighbour metrics: kBET, the metric used to evaluate batch correction methods in biology, and a local Silhouette Coefficient s_{k,c}. In both cases a low value close to zero indicates good local mixing of sample representations.

## Citation

```bibtex
@misc{foster2021comp,
  title={Contrastive Mixture of Posteriors for Counterfactual Inference, Data Integration and Fairness},
  author={Foster et al. (2021)},
  year={2021},
  note={arXiv:2106.08161}
}
```

- arXiv: 2106.08161