Cimi4d Annotation Eval

---
name: cimi4d-annotation-eval
description: Probes the accuracy of automatically generated 3D human pose and translation annotations for rock climbing motions. It evaluates how well a LiDAR-IMU fusion and blending optimization pipeline reconstructs off-ground climbing poses compared to manual ground truth. Use when the user wants to benchmark on CIMI4D, or asks about evaluating this task. Reports PMPJPE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.17948
  bibtex_key: yan2023cimi4d
  confidence: high
---

# cimi4d-annotation-eval

> CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions — Ming Yan et al. (2023) (arXiv:2303.17948, 2023)

## What this evaluates

Probes the accuracy of automatically generated 3D human pose and translation annotations for rock climbing motions. It evaluates how well a LiDAR-IMU fusion and blending optimization pipeline reconstructs off-ground climbing poses compared to manual ground truth.

## Datasets

- **CIMI4D** — total 180000; splits: eval (-1)

## Metrics

- `PMPJPE` **(primary)** — range: other
  - Procrustes-Aligned Mean Per Joint Position Error. Computed by first aligning predicted and ground-truth joint sets via rigid Procrustes analysis, then averaging the Euclidean distances across all joints. Measured in millimeters.
- `MPJPE` — range: other
  - Mean Per Joint Position Error. Computed as the average Euclidean distance between predicted and ground-truth joint positions without alignment. Measured in millimeters.
- `PCK0.5` — range: [0, 1]
  - Percentage of Correct Keypoints. The fraction of predicted joints whose distance to the ground-truth joint is below a threshold of 0.5. Reported as a ratio.
- `PVE` — range: other
  - Per Vertex Error. The average Euclidean distance between corresponding vertices of predicted and ground-truth 3D meshes. Measured in millimeters.
- `ACCEL` — range: other
  - Acceleration Error. The average difference in joint acceleration between predicted and ground-truth sequences. Measured in m/s².

## Input / output format

**Input**: Synchronized RGB images, LiDAR point clouds, IMU pose/translation data, and high-precision static point clouds of climbing walls.

**Output**: Reconstructed 3D human joint positions and body translation, evaluated against manually annotated ground truth poses.

## Scoring recipe

```python
def compute_mpjpe(pred, gt):
    return np.mean(np.linalg.norm(pred - gt, axis=2))
def compute_pmpjpe(pred, gt):
    aligned_pred = procrustes_align(pred, gt)
    return np.mean(np.linalg.norm(aligned_pred - gt, axis=2))
def compute_pve(pred_mesh, gt_mesh):
    return np.mean(np.linalg.norm(pred_mesh - gt_mesh, axis=1))
def compute_accel(pred_j, gt_j):
    pred_a = np.diff(pred_j, axis=0, n=2)
    gt_a = np.diff(gt_j, axis=0, n=2)
    return np.mean(np.linalg.norm(pred_a - gt_a, axis=2))
```

## Common pitfalls

- ACCEL is reported in m/s² while all other error metrics (PMPJPE, MPJPE, PVE) are in millimeters.
- PMPJPE requires Procrustes alignment of predicted and ground-truth poses before computing joint errors, unlike raw MPJPE.
- The reported metrics evaluate the quality of the dataset's annotation pipeline rather than downstream task performance like pose prediction or generation.

## Evidence (verbatim from paper)

> Evaluation metrics. In this section and in Sec. 5, we report Procrustes-Aligned Mean Per Joint Position Error (PMPJPE), Mean Per Joint Position Error (MPJPE), Percentage of Correct Keypoints (PCK), Per Vertex Error (PVE), and Acceleration error(m/s^{2}) (ACCEL). Except ACCEL, error metrics are measured in millimeters.

## Citation

```bibtex
@misc{yan2023cimi4d,
  title={CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions},
  author={Ming Yan et al. (2023)},
  year={2023},
  note={arXiv:2303.17948}
}
```

- arXiv: 2303.17948