Unsupervised Near Duplicate Eval

---
name: unsupervised-near-duplicate-eval
description: Evaluates the ability of image descriptors to distinguish near-duplicate image pairs from non-duplicates under extreme specificity constraints, simulating large-scale forensic or fraud detection scenarios. Use when the user wants to benchmark on MFND (Mir-Flickr Near-Duplicate), CLAIMS, Holidays, California-ND, or asks about evaluating this task. Reports sensitivity at false positive rate (FPR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1907.02821
  bibtex_key: morra2019benchmarking
  confidence: high
---

# unsupervised-near-duplicate-eval

> Benchmarking unsupervised near-duplicate image detection — Morra et al. (2019) (arXiv:1907.02821, 2019)

## What this evaluates

Evaluates the ability of image descriptors to distinguish near-duplicate image pairs from non-duplicates under extreme specificity constraints, simulating large-scale forensic or fraud detection scenarios.

## Datasets

- **MFND (Mir-Flickr Near-Duplicate)** — total ?; splits: test (-1)
- **CLAIMS** — total ?; splits: test (-1)
- **Holidays** — total ?; splits: test (-1)
- **California-ND** — total ?; splits: test (-1)

## Metrics

- `sensitivity at false positive rate (FPR)` **(primary)** — range: percent
  - Sensitivity = TP / (TP + FN). Evaluated at a fixed, extremely low FPR threshold (e.g., 10^-6 to 10^-9) to ensure high specificity for forensic applications.

## Input / output format

**Input**: Single images processed through pre-trained or fine-tuned CNNs/global descriptors to extract fixed-length feature vectors. Evaluation is performed on image pairs (query, candidate) to compute similarity scores.

**Output**: Binary prediction (near-duplicate vs. non-duplicate) or similarity score used to rank candidates.

## Scoring recipe

```python
def compute_sensitivity_at_fpr(predictions, gold, fpr_threshold):
    tp = sum(1 for p, g in zip(predictions, gold) if p >= threshold and g == 1)
    fn = sum(1 for p, g in zip(predictions, gold) if p < threshold and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p >= threshold and g == 0)
    tn = sum(1 for p, g in zip(predictions, gold) if p < threshold and g == 0)
    sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
    return sensitivity if fpr <= fpr_threshold else None
```

## Common pitfalls

- Hard negative mining can introduce label noise; the paper found ~2-2.4% actual near-duplicates in the negative mining pools that were relabeled.
- Standard accuracy or AUC-ROC can be misleading due to extreme class imbalance and the requirement for ultra-high specificity (10^-9); sensitivity at a fixed low FPR is the correct metric.
- Using different PCA whitening parameters trained on different subsets can cause data leakage if not strictly separated from the test set.

## Evidence (verbatim from paper)

> reframes the problem as a binary classification task via ROC analysis, showing that fine-tuned CNNs outperform off-the-shelf features, with MFND achieving 96% sensitivity at a false positive rate of 1.43e-6

## Citation

```bibtex
@misc{morra2019benchmarking,
  title={Benchmarking unsupervised near-duplicate image detection},
  author={Morra et al. (2019)},
  year={2019},
  note={arXiv:1907.02821}
}
```

- arXiv: 1907.02821