Mammo Concept Alignment Eval

---
name: mammo-concept-alignment-eval
description: Evaluates how well vision-language models and CNNs capture clinically relevant mammography concepts at the neuron level. It quantifies concept coverage, alignment strength, and how domain-specific pretraining or task-specific fine-tuning shifts learned representations. Use when the user wants to benchmark on VinDR-Mammo, EMBED, or asks about evaluating this task. Reports unique_concepts_captured.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.21102
  bibtex_key: salahuddin2025mammo
  confidence: high
---

# mammo-concept-alignment-eval

> Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models — Salahuddin et al. (2025) (arXiv:2509.21102, 2025)

## What this evaluates

Evaluates how well vision-language models and CNNs capture clinically relevant mammography concepts at the neuron level. It quantifies concept coverage, alignment strength, and how domain-specific pretraining or task-specific fine-tuning shifts learned representations.

## Datasets

- **VinDR-Mammo** — total ?; splits: test (-1); repo https://github.com/Suaiba/Mammo-CLIP-Dissect
- **EMBED** — total 84; splits: probe (84); repo https://github.com/Suaiba/Mammo-CLIP-Dissect

## Metrics

- `unique_concepts_captured` **(primary)** — range: other
  - Counts the number of distinct mammography concepts whose mean similarity score to a target neuron exceeds a predefined threshold $	au$ (typically the layer-specific or global mean similarity). Overlapping concept terms are deduplicated to prevent double-counting.
- `mean_similarity_score` — range: [0, 1]
  - Similarity metric (e.g., cosine) between neuron activation vectors from the target model ($F_{target}$) and concept embeddings from the dissector model ($F_{dissector}$). Used to establish the threshold $	au$ for concept capture.

## Input / output format

**Input**: Mammography images from probe datasets (VinDR-Mammo test set or EMBED subset) passed through a target CNN/VLM feature extractor ($F_{target}$) to obtain neuron activations, which are then compared against a fixed vocabulary of mammography concept embeddings using a vision-language model ($F_{dissector}$).

**Output**: Per-neuron similarity scores against all concepts in the vocabulary, plus a binary determination of whether each concept surpasses the threshold $	au$. Aggregated counts of unique concepts per layer and model variant.

## Scoring recipe

```python
def score_concept_coverage(neuron_activations, concept_embeddings, threshold_type='mean'):
    similarities = cosine_similarity(neuron_activations, concept_embeddings)
    if threshold_type == 'mean':
        tau = np.mean(similarities)
    else:
        tau = threshold_type
    captured = [c for c, s in zip(concepts, similarities) if s > tau]
    captured = deduplicate_overlapping_terms(captured)
    return len(captured)
```

## Common pitfalls

- Threshold $	au$ is often set to the mean similarity, which varies significantly across layers and models, making cross-model comparisons highly sensitive to this choice.
- The concept vocabulary contains overlapping terms (e.g., 'extremely' vs 'extremely dense'); failure to explicitly deduplicate inflates or deflates coverage counts.
- Neuron-level responsiveness is heavily dependent on probe dataset composition (e.g., EMBED's implant subset vs VinDR-Mammo test set), limiting the generalizability of single-neuron examples.

## Evidence (verbatim from paper)

> First, we compare the mean similarity scores between neurons and concepts, used as thresholds, $	au$, at each layer within both $F_{target}$ models. This analysis is presented in Figure [4(a)]. From this figure, we can observe that M-Mammo-CLIP Dissect consistently has higher $	au$ values than G-Mammo-CLIP Dissect. Next, we analyse the number of unique concepts captured, surpassing the layer-specific threshold $	au$ across three representative layers.

## Citation

```bibtex
@misc{salahuddin2025mammo,
  title={Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models},
  author={Salahuddin et al. (2025)},
  year={2025},
  note={arXiv:2509.21102}
}
```

- arXiv: 2509.21102