Poi Eval

---
name: poi-eval
description: Probes a model's ability to identify privacy-sensitive objects in images by reasoning about scene context rather than relying solely on visual appearance. It evaluates whether the system can distinguish between obvious privacy leaks (e.g., faces) and context-dependent sensitive information (e.g., people in specific roles). Use when the user wants to benchmark on MOSAIC, PRIVACY1000, or asks about evaluating this task. Reports F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.12736
  bibtex_key: jiang2024beyond
  confidence: high
---

# poi-eval

> Beyond Visual Appearances: Privacy-sensitive Objects Identification via Hybrid Graph Reasoning — Jiang et al. (2024) (arXiv:2406.12736, 2024)

## What this evaluates

Probes a model's ability to identify privacy-sensitive objects in images by reasoning about scene context rather than relying solely on visual appearance. It evaluates whether the system can distinguish between obvious privacy leaks (e.g., faces) and context-dependent sensitive information (e.g., people in specific roles).

## Datasets

- **MOSAIC** — total 13384; splits: test (13384)
- **PRIVACY1000** — total 1000; splits: train (800), val (200)

## Metrics

- `F1 Score` **(primary)** — range: [0, 1]
  - Harmonic mean of Precision and Recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Computed per dataset split.
- `Precision` — range: [0, 1]
  - Ratio of true positive detections to all positive detections: Precision = TP / (TP + FP).
- `Recall` — range: [0, 1]
  - Ratio of true positive detections to all actual positives: Recall = TP / (TP + FN).

## Input / output format

**Input**: RGB images containing scenes with potential privacy-sensitive objects.

**Output**: Bounding box coordinates and a binary privacy-sensitive classification label for each detected object.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if iou(p.box, g.box) > 0.5 and p.label == g.label)
    fp = len(preds) - tp
    fn = len(gold) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return {'precision': prec, 'recall': rec, 'f1_score': f1}
```

## Common pitfalls

- PRIVACY1000 is explicitly stated as not publicly available, hindering direct reproduction.
- Privacy sensitivity is subjective; annotations use a majority-rule approach across multiple annotators.
- Models must perform contextual reasoning, not just visual detection, as evidenced by YOLOv5's failure on context-dependent objects.

## Evidence (verbatim from paper)

> Table 1: Experimental results for the two privacy datasets using different algorithms.  

<table><tr><td colspan="4">PRIVACY1000</td><td colspan="3">MOSAIC</td></tr><tr><td>Methods</td><td>Precision</td><td>Recall</td><td>F1 Score</td><td>Precision</td><td>Recall</td><td>F1 Score</td></tr>

## Citation

```bibtex
@misc{jiang2024beyond,
  title={Beyond Visual Appearances: Privacy-sensitive Objects Identification via Hybrid Graph Reasoning},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2406.12736}
}
```

- arXiv: 2406.12736