Rp 1k Eval

---
name: rp-1k-eval
description: Evaluates a model's ability to discover recurring visual patterns in a single image. It measures detection accuracy at both the individual pattern instance level and the whole pattern level against human annotations. Use when the user wants to benchmark on RP-1K, or asks about evaluating this task. Reports RP Instance Recall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.07991
  bibtex_key: zhang2022novel3d
  confidence: high
---

# rp-1k-eval

> Novel 3D Scene Understanding Applications From Recurrence in a Single Image — Zhang et al. (2022) (arXiv:2210.07991, 2022)

## What this evaluates

Evaluates a model's ability to discover recurring visual patterns in a single image. It measures detection accuracy at both the individual pattern instance level and the whole pattern level against human annotations.

## Datasets

- **RP-1K** — total 1024; splits: test (1024)

## Metrics

- `RP Instance Precision` — range: [0, 1]
  - Precision at the instance level: $P_I = |\mathbf{RPI_A}| / |\mathbf{RPI_D}|$, where $\mathbf{RPI_A}$ are detected instances accepted via IOD > 0.5, and $\mathbf{RPI_D}$ are all detected instances.
- `RP Instance Recall` **(primary)** — range: [0, 1]
  - Recall at the instance level: $R_I = |\mathbf{RPI_A}| / |\mathbf{RPI_{GT}}|$, where $\mathbf{RPI_{GT}}$ are all ground truth instances.
- `RP Precision` — range: [0, 1]
  - Precision at the pattern level: $P_{RP} = |\mathbf{RP_A}| / |\mathbf{RP_D}|$, where $\mathbf{RP_A}$ are detected patterns assigned to a ground truth pattern based on highest instance precision.
- `RP Recall` — range: [0, 1]
  - Recall at the pattern level: $R_{RP} = |\mathbf{RP_A}| / |\mathbf{RP_{GT}}|$, where $\mathbf{RP_{GT}}$ are all ground truth patterns.

## Input / output format

**Input**: Single RGB image.

**Output**: A set of detected Recurring Pattern Instances (RPIs) and a set of detected Recurring Patterns (RPs) grouping these instances.

## Scoring recipe

```python
def compute_metrics(pred_rpis, gt_rpis, h=0.5):
    accepted = []
    for p in pred_rpis:
        for g in gt_rpis:
            if intersection_area(p, g) / area(p) > h:
                accepted.append(p)
                break
    p_inst = len(accepted) / len(pred_rpis) if pred_rpis else 0
    r_inst = len(accepted) / len(gt_rpis) if gt_rpis else 0
    return p_inst, r_inst
```

## Common pitfalls

- Uses Intersection-over-Detection (IOD) instead of standard IoU to evaluate partial overlaps of recurring patterns.
- RP-level matching assigns a detected RP to the ground truth RP with the highest instance-level precision, not IoU or recall.
- The IOD threshold h=0.5 is fixed for reported results; altering it changes acceptance rates significantly.

## Evidence (verbatim from paper)

> We propose an intersection-over-detection (IOD) metric as follows. If $RPI_{i}$ is a detected RPI, and $RPI_{GT_{j}}$ a groundtruth RPI. $RPI_{j}$ is considered acceptable if and only if $(RPI_{i}\cap RPI_{GT_{j}})/RPI_{i}>h$, where $h$ is a numerical threshold. Given a detected RP with a set of RPIs $\mathbf{RPI_{D}}$, and some ground truth RP with a set of ground truth RPIs $\mathbf{RPI_{GT}}$, and the set of acceptable RPIs denoted as $\mathbf{RPI_{A}}$, RP Instance level precision $P_{I}$ and recall $R_{I}$ rates are defined as: $P_{I}\=|\mathbf{RPI_{A}}|/|\mathbf{RPI_{D}}|$, $R_{I}\=|\mathbf{RPI_{A}}|/|\mathbf{RPI_{GT}}|$

## Citation

```bibtex
@misc{zhang2022novel3d,
  title={Novel 3D Scene Understanding Applications From Recurrence in a Single Image},
  author={Zhang et al. (2022)},
  year={2022},
  note={arXiv:2210.07991}
}
```

- arXiv: 2210.07991