Hoi Detection Eval

---
name: hoi-detection-eval
description: Evaluates a model's ability to detect Human-Object Interactions (HOIs) by predicting triplets of person, verb, and object along with their bounding boxes. It specifically probes the model's robustness to object bias by measuring performance on rare versus frequent interactions under both standard and object-conditional evaluation protocols. Use when the user wants to benchmark on HICO-DET, HOI-COCO, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.02400
  bibtex_key: wang2022chairs
  confidence: high
---

# hoi-detection-eval

> Chairs Can be Stood on: Overcoming Object Bias in Human-Object Interaction Detection — Wang et al. (2022) (arXiv:2207.02400, 2022)

## What this evaluates

Evaluates a model's ability to detect Human-Object Interactions (HOIs) by predicting triplets of person, verb, and object along with their bounding boxes. It specifically probes the model's robustness to object bias by measuring performance on rare versus frequent interactions under both standard and object-conditional evaluation protocols.

## Datasets

- **HICO-DET** — total 47776; splits: train (38118), test (9658)
- **HOI-COCO** — total 9915; splits: train (4969), test (4946)

## Metrics

- `mAP` **(primary)** — range: percent
  - Mean Average Precision across HOI classes. A prediction is positive if the HOI class matches ground truth and both human and object bounding box IoUs exceed 0.5. Reported separately for full, rare (<10 training instances), and non-rare classes.
- `OR` — range: percent
  - Mean Average Precision for Object-Rare classes. An interaction is object-rare if N_v,o / N_o < 0.3. Computed per object, then averaged across all objects.
- `ONR` — range: percent
  - Mean Average Precision for Object-NonRare classes. An interaction is object-non-rare if N_v,o / N_o >= 0.3. Computed per object, then averaged across all objects.
- `AVE` — range: percent
  - Average of OR and ONR scores, providing a single summary metric for the object-bias evaluation protocol.

## Input / output format

**Input**: RGB images with annotated human and object bounding boxes, and HOI triplets (person, verb, object).

**Output**: Predicted HOI triplets with corresponding human and object bounding boxes, along with confidence scores for each triplet.

## Scoring recipe

```python
def compute_hoi_metrics(predictions, ground_truth, iou_thresh=0.5):
    matches = []
    for pred in predictions:
        for gt in ground_truth:
            if pred.hoi_class == gt.hoi_class:
                if (iou(pred.human_box, gt.human_box) > iou_thresh and
                    iou(pred.obj_box, gt.obj_box) > iou_thresh):
                    matches.append((pred, gt))
                    break
    aps = {cls: compute_ap(matches_for_cls) for cls in unique_classes}
    return {'mAP_full': mean(aps.values()),
            'mAP_rare': mean(aps[c] for c in rare_classes),
            'mAP_non_rare': mean(aps[c] for c in non_rare_classes)}
```

## Common pitfalls

- Confusing the standard rare/non-rare split (<10 training instances) with the object-bias split (N_v,o / N_o < 0.3).
- Forgetting that both human and object bounding boxes must independently exceed 0.5 IoU with ground truth for a positive match.
- Comparing results across different detector settings (pre-trained, fine-tuned, oracle) without noting that detection quality heavily influences HOI scores.

## Evidence (verbatim from paper)

> Standard Evaluation Metrics We followed the standard evaluation setting [4] and reported mean average precision (mAP) for both datasets, where the mAP on rare (less than 10 training instances), non-rare and full classes are reported. For both settings, a prediction is regarded as positive if (1) the HOI classification is correct and (2) the detected human and object bounding boxes have IoUs greater than 0.5 with the ground-truth bounding box.

## Citation

```bibtex
@misc{wang2022chairs,
  title={Chairs Can be Stood on: Overcoming Object Bias in Human-Object Interaction Detection},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2207.02400}
}
```

- arXiv: 2207.02400