Cbm Concept Accuracy Eval

---
name: cbm-concept-accuracy-eval
description: Evaluates whether Concept Bottleneck Models learn semantically meaningful concept representations from input images under varying annotation granularity and concept correlation structures. Measures how well the model predicts intermediate concepts and downstream tasks compared to standard neural networks. Use when the user wants to benchmark on Playing cards, CheXpert, or asks about evaluating this task. Reports concept accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.00912
  bibtex_key: furby2024constrain
  confidence: high
---

# cbm-concept-accuracy-eval

> Can we Constrain Concept Bottleneck Models to Learn Semantically Meaningful Input Features? — Furby et al. (2024) (arXiv:2402.00912, 2024)

## What this evaluates

Evaluates whether Concept Bottleneck Models learn semantically meaningful concept representations from input images under varying annotation granularity and concept correlation structures. Measures how well the model predicts intermediate concepts and downstream tasks compared to standard neural networks.

## Datasets

- **Playing cards** — total 40000; splits: train (28000), val (12000)
- **CheXpert** — total 224316; splits: train (-1), val (202)

## Metrics

- `concept accuracy` **(primary)** — range: [0, 1]
  - Average accuracy across all predicted concepts for a given dataset variation. Calculated as the proportion of correctly predicted concept labels divided by the total number of concept predictions.
- `AUC` — range: [0, 1]
  - Area Under the receiver operating characteristic Curve for the downstream task prediction (e.g., classifying hand ranks or 'no_findings').

## Input / output format

**Input**: RGB images of playing cards (single, three random, or three poker cards) or chest X-rays (512x512 pixels).

**Output**: Binary concept predictions (instance-level or class-level) and a downstream task prediction (hand rank classification or 'no_findings' binary label).

## Scoring recipe

```python
def compute_concept_accuracy(preds, golds):
    correct = sum(1 for p, g in zip(preds, golds) if p == g)
    return correct / len(golds)

def compute_auc(preds, golds):
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(golds, preds)
```

## Common pitfalls

- Concept accuracy is averaged across all concepts in the dataset variation, not computed per-image or per-concept separately.
- CheXpert validation uses only 202 frontal-view images from the official splits, not the full validation set.
- Class-level concept modifications replace instance annotations with the most common concept vector for 3, 4, or 5 concepts present, altering the ground truth distribution.

## Evidence (verbatim from paper)

> Our random cards models achieve an average concept accuracy of 99.932%, poker card models have an average concept accuracy of 99.914% and class-level poker cards have an average concept accuracy of 99.99%.

## Citation

```bibtex
@misc{furby2024constrain,
  title={Can we Constrain Concept Bottleneck Models to Learn Semantically Meaningful Input Features?},
  author={Furby et al. (2024)},
  year={2024},
  note={arXiv:2402.00912}
}
```

- arXiv: 2402.00912