Posebench Robustness Eval

---
name: posebench-robustness-eval
description: Evaluates the robustness of human and animal pose estimation models when subjected to real-world image corruptions such as blur, noise, compression, lighting changes, and occlusion masks. It measures how much model accuracy degrades relative to clean-image performance across varying corruption severities. Use when the user wants to benchmark on COCO-C, OCHuman-C, AP10K-C, or asks about evaluating this task. Reports mRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.14367
  bibtex_key: ma2024posebench
  confidence: high
---

# posebench-robustness-eval

> PoseBench: Benchmarking the Robustness of Pose Estimation Models under Corruptions — Ma et al. (2024) (arXiv:2406.14367, 2024)

## What this evaluates

Evaluates the robustness of human and animal pose estimation models when subjected to real-world image corruptions such as blur, noise, compression, lighting changes, and occlusion masks. It measures how much model accuracy degrades relative to clean-image performance across varying corruption severities.

## Datasets

- **COCO-C** — total 5000; splits: val (5000)
- **OCHuman-C** — total 5081; splits: val (5081)
- **AP10K-C** — total 10015; splits: val (10015)

## Metrics

- `mAP` — range: [0, 1]
  - Mean Average Precision, computed as the average of Average Precision (AP) scores at IoU thresholds from 0.5 to 0.95 with a step of 0.05, following COCO keypoint evaluation standards.
- `mAR` — range: [0, 1]
  - Mean Average Recall, computed analogously to mAP but measuring the maximum recall achievable across different numbers of detected instances per image.
- `mRR` **(primary)** — range: [0, 1]
  - Mean Relative Robustness, defined as the average of relative robustness scores across C corruption types. For each corruption c, RR_c is the mean over 5 severity levels of (1 - (mAP_clean - mAP_c,s) / mAP_clean), which simplifies to the ratio of corrupted mAP to clean mAP.

## Input / output format

**Input**: Corrupted images (10 corruption types × 5 severity levels) with ground truth keypoint annotations for each person/animal instance.

**Output**: Predicted keypoint coordinates (x, y) and confidence scores for each detected instance.

## Scoring recipe

```python
def compute_mRR(pred_clean, gt_clean, pred_corrupted, gt_corrupted):
    mAP_clean = compute_coco_mAP(pred_clean, gt_clean)
    mAP_corrupted = compute_coco_mAP(pred_corrupted, gt_corrupted)
    # Relative robustness for a specific corruption/severity
    rr = mAP_corrupted / mAP_clean
    return rr
# mRR = mean over 5 severity levels per corruption, then mean over C corruptions
```

## Common pitfalls

- mRR is a relative ratio (higher is better), not an absolute performance drop. Interpreting it as a percentage decrease will invert the ranking.
- Must average over all 5 severity levels for each corruption type before averaging across the 10 corruption types, as specified in the formula.
- Uses COCO-style mAP (average of AP@[.5:.95]), not just AP@0.5.

## Evidence (verbatim from paper)

> Following[[48]], we introduce the robustness metric, mean Relative Robustness (mRR), to evaluate how much a model’s performance drops under certain corruptions compared to clean images. To calculate this metric, we first evaluate the model on clean images and obtain the mean Average Precision (mAP), denoted as $mAP_{clean}$. For any corruption $c$, we then calculate the mAP at each severity level $s$, denoted as $mAP_{c,s}$. The relative robustness for corruption $c$ and the mean Relative Robustness (mRR) are defined as follows: $RR_{c}=\frac{1}{5}\sum_{s=1}^{5}(1-\frac{mAP_{clean}-mAP_{c,s}}{mAP_{clean}}), \ \ mRR=\frac{1}{C}\sum_{c=1}^{C}RR_{c}$ where $c$ indexes the $C$ types of corruption.

## Citation

```bibtex
@misc{ma2024posebench,
  title={PoseBench: Benchmarking the Robustness of Pose Estimation Models under Corruptions},
  author={Ma et al. (2024)},
  year={2024},
  note={arXiv:2406.14367}
}
```

- arXiv: 2406.14367