Privacybench Eval

---
name: privacybench-eval
description: Evaluates the trade-offs between privacy preservation, model utility, and computational/energy costs in hybrid privacy-preserving vision systems. It probes how combining federated learning with differential privacy or secure multi-party computation affects convergence, classification accuracy, and resource consumption across different neural architectures. Use when the user wants to benchmark on Alzheimer MRI Classification, ISIC Skin Lesion Classification, or asks about evaluating this task. Reports MCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.18900
  bibtex_key: obiefuna2026privacybench
  confidence: high
---

# privacybench-eval

> PrivacyBench: Privacy Isn't Free in Hybrid Privacy-Preserving Vision Systems — Obiefuna et al. (2026) (arXiv:2602.18900, 2026)

## What this evaluates

Evaluates the trade-offs between privacy preservation, model utility, and computational/energy costs in hybrid privacy-preserving vision systems. It probes how combining federated learning with differential privacy or secure multi-party computation affects convergence, classification accuracy, and resource consumption across different neural architectures.

## Datasets

- **Alzheimer MRI Classification** — total ?; splits: train (-1), val (-1)
- **ISIC Skin Lesion Classification** — total ?; splits: train (-1), val (-1)

## Metrics

- `MCC` **(primary)** — range: [-1, 1]
  - Matthews Correlation Coefficient measures the quality of binary/multiclass classifications. It returns a value between -1 and +1, where +1 is perfect prediction, 0 is random prediction, and -1 is inverse prediction.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified instances out of the total number of instances.

## Input / output format

**Input**: Preprocessed MRI brain scans or dermoscopic images fed into ResNet18 or ViT-Base architectures.

**Output**: Class predictions (4 classes for Alzheimer MRI, 8 classes for ISIC Skin Lesion) with associated confidence scores or logits.

## Scoring recipe

```python
def evaluate(y_true, y_pred):
    accuracy = np.mean(y_true == y_pred)
    mcc = matthews_corrcoef(y_true, y_pred)  # Multiclass-aware per sklearn convention
    return {'accuracy': accuracy, 'mcc': mcc}
```

## Common pitfalls

- Assuming privacy techniques combine additively; the benchmark shows hybrid combinations (e.g., FL+DP) cause non-additive convergence failures and compounding overhead.
- Evaluating privacy methods in isolation rather than testing hybrid configurations (FL+DP, FL+SMPC) that reflect real-world deployment constraints.
- Focusing solely on accuracy while ignoring critical resource metrics like energy consumption (kWh) and memory utilization, which determine deployment feasibility.

## Evidence (verbatim from paper)

> Classification Performance: Standard metrics including accuracy, F1-score, Matthews Correlation Coefficient (MCC), precision, recall, and ROC-AUC, with particular emphasis on MCC for robust evaluation of privacy technique impacts on learning quality*[[12] over f1 score and accuracy in binary classification evaluation")]*.

## Citation

```bibtex
@misc{obiefuna2026privacybench,
  title={PrivacyBench: Privacy Isn't Free in Hybrid Privacy-Preserving Vision Systems},
  author={Obiefuna et al. (2026)},
  year={2026},
  note={arXiv:2602.18900}
}
```

- arXiv: 2602.18900