Mmd Js Divergence

---
name: mmd_js_divergence
description: Evaluates the geometric alignment between visual and textual attention key vectors in multimodal LLMs. It probes whether visual inputs occupy an out-of-distribution subspace relative to the text-centric key space learned during pretraining. Use when the user has predictions and gold and needs to compute MMD.
metadata:
  skill_kind: metric
  source_arxiv: 2510.26721
  bibtex_key: zheng2025unveiling
  confidence: high
---

# mmd_js_divergence

> Unveiling Intrinsic Text Bias in Multimodal Large Language Models through Attention Key-Space Analysis — Zheng et al. (2025) (arXiv:2510.26721, 2025)

## What this evaluates

Evaluates the geometric alignment between visual and textual attention key vectors in multimodal LLMs. It probes whether visual inputs occupy an out-of-distribution subspace relative to the text-centric key space learned during pretraining.

## Datasets

- **MMBench-CN** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)

## Metrics

- `MMD` **(primary)** — range: other
  - Maximum Mean Discrepancy measuring the distance between mean embeddings of image and text key vectors in a reproducing kernel Hilbert space. Computed on PCA-reduced key vectors.
- `JS` — range: [0, 1]
  - Jensen-Shannon divergence measuring the similarity between the probability distributions of image and text key vectors after PCA embedding. Values range from 0 (identical) to 1 (maximally different).

## Input / output format

**Input**: Paired image and text tokens processed through the decoder layers of an MLLM (LLaVA-1.5-7B or Qwen2.5-VL-7B) to extract attention key vectors at specified layers.

**Output**: Scalar divergence scores (MMD and JS) quantifying the distributional gap between visual and textual key vectors, aggregated per layer and benchmark.

## Scoring recipe

```python
# Extract key vectors for image tokens (K_img) and text tokens (K_txt) at layer l
# Apply PCA to reduce dimensionality
K_img_pca = pca.fit_transform(K_img)
K_txt_pca = pca.fit_transform(K_txt)
# Compute MMD (using RBF kernel or similar as per paper)
mmd_score = compute_mmd(K_img_pca, K_txt_pca)
# Compute JS divergence (requires density estimation or histogram binning)
js_score = compute_js_divergence(K_img_pca, K_txt_pca)
return mmd_score, js_score
```

## Common pitfalls

- MMD and JS values are sensitive to the choice of kernel and PCA dimensionality; results should not be compared across different preprocessing settings.
- The metric measures representation alignment, not downstream task accuracy; high divergence does not necessarily imply poor model performance.
- Intra-modality controls (Image vs Image, Text vs Text) must be computed to distinguish structural bias from measurement noise.

## Evidence (verbatim from paper)

> Using the PCA embeddings, we compute MMD- and JS-based modality divergence for each layer and benchmark. The aggregated statistical analysis confirms the core hypothesis by demonstrating a vast separation between cross-modality and intra-modality comparisons: The mean MMD for the Cross-Modality Gap (Image V.S. Text) is 0.408 (std=0.346), with the maximum divergence reaching 1.054 (LLaVA-1.5B, Layer 2).

## Citation

```bibtex
@misc{zheng2025unveiling,
  title={Unveiling Intrinsic Text Bias in Multimodal Large Language Models through Attention Key-Space Analysis},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2510.26721}
}
```

- arXiv: 2510.26721