Disc21 Eval

---
name: disc21-eval
description: This benchmark evaluates image copy detection systems under realistic, adversarial conditions. It probes a model's ability to match transformed query images against a large reference database while resisting geometric, color, overlay, and deepfake manipulations. The setup emphasizes scalability and robustness in a high-false-positive-rate, needle-in-haystack search regime. Use when the user wants to benchmark on DISC21, or asks about evaluating this task. Reports micro Average Precision.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.09672
  bibtex_key: douze2021disc21
  confidence: high
---

# disc21-eval

> The 2021 Image Similarity Dataset and Challenge — Douze et al. (2021) (arXiv:2106.09672, 2021)

## What this evaluates

This benchmark evaluates image copy detection systems under realistic, adversarial conditions. It probes a model's ability to match transformed query images against a large reference database while resisting geometric, color, overlay, and deepfake manipulations. The setup emphasizes scalability and robustness in a high-false-positive-rate, needle-in-haystack search regime.

## Datasets

- **DISC21** — total 2100000; splits: train (1000000), reference (1000000), dev (50000), test (50000); repo https://github.com/facebookresearch/isc2021

## Metrics

- `micro Average Precision` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve computed jointly across all queries. Formula: μAP = Σ p(i)Δr(i), where p(i) is precision at rank i, Δr(i) is the recall increment, and N is the total number of returned pairs.

## Input / output format

**Input**: A query image (subjected to manual or automatic transformations) and a reference database of 1 million source images.

**Output**: A ranked list of pairs (query image, candidate source image from reference set) accompanied by a confidence score. Models may omit queries that are not matched.

## Scoring recipe

```python
def compute_micro_ap(predictions, gold_positives, total_positives=10000):
    sorted_preds = sorted(predictions, key=lambda x: x[2], reverse=True)
    tp, fp, ap_sum, prev_recall = 0, 0, 0.0, 0.0
    for q, r, _ in sorted_preds:
        if (q, r) in gold_positives:
            tp += 1
        else:
            fp += 1
        precision = tp / (tp + fp)
        recall = tp / total_positives
        ap_sum += precision * (recall - prev_recall)
        prev_recall = recall
    return ap_sum
```

## Common pitfalls

- Confusing micro-AP with macro-AP (mAP); micro-AP computes a single PR curve over all queries, while macro-AP averages AP per query.
- Ignoring distractor queries; false positives from distractor queries directly penalize precision and thus the micro-AP score.
- Assuming all queries must have a match; the protocol explicitly allows models to return empty lists for distractor queries.

## Evidence (verbatim from paper)

> We use micro Average Precision to measure performance... It is computed as μAP=∑_{i=1}^{N}p(i)Δr(i)∈[0,1] where p(i) is the precision at position i of the sorted list of pairs, Δr(i) is the difference of recall between position i and i-1, and N is the total number of returned pairs for all queries.

## Citation

```bibtex
@misc{douze2021disc21,
  title={The 2021 Image Similarity Dataset and Challenge},
  author={Douze et al. (2021)},
  year={2021},
  note={arXiv:2106.09672}
}
```

- arXiv: 2106.09672