Manip Eval

---
name: manip-eval
description: Evaluates physically-grounded image editing by measuring 2D spatial accuracy, depth prediction, 3D geometric consistency, image quality, and VLM-based physical plausibility for object manipulation tasks. Use when the user wants to benchmark on ManipEval, or asks about evaluating this task. Reports Chamfer.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.07230
  bibtex_key: xu2026phyedit
  confidence: high
---

# manip-eval

> PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing — Xu et al. (2026) (arXiv:2604.07230, 2026)

## What this evaluates

Evaluates physically-grounded image editing by measuring 2D spatial accuracy, depth prediction, 3D geometric consistency, image quality, and VLM-based physical plausibility for object manipulation tasks.

## Datasets

- **ManipEval** — total 200; splits: test (200)

## Metrics

- `DIoU` — range: [0, 100]
  - Distance IoU between predicted and ground-truth bounding boxes.
- `Mask IoU` — range: [0, 100]
  - Intersection over Union between predicted and ground-truth object masks.
- `AbsRel` — range: [0, 100]
  - Mean relative absolute difference over valid object pixels: (1/|Ω|) Σ |D_pred - D_gt| / D_gt.
- `δ1.25` — range: [0, 100]
  - Ratio of pixels satisfying max(D_pred/D_gt, D_gt/D_pred) < 1.25.
- `Chamfer` **(primary)** — range: [0, 100]
  - Symmetric Chamfer Distance between predicted and ground-truth point clouds, normalized by the valid-scene diagonal.
- `Centroid` — range: [0, 100]
  - L2 distance between predicted and ground-truth point cloud centroids.
- `RA-DINO` — range: [0, 100]
  - DINO similarity penalized by relocation-vector errors: S_DINO * exp(-α e_∥ - β e_⊥), with α=1, β=0.8.
- `DeQA` — range: [0, 100]
  - General perceptual quality metric score.
- `Phys-VLM` — range: [0, 100]
  - VLM-based assessment of physical realism and global scene consistency (lighting/shadows, depth ordering, contacts/occlusions).

## Input / output format

**Input**: Source image, spatial trajectory/instructions, and optionally a 3D-transformed reference image.

**Output**: Edited image(s) with manipulated objects, predicted depth map, and reconstructed object point clouds.

## Scoring recipe

```python
def compute_metrics(pred_img, gt_img, pred_depth, gt_depth, pred_pc, gt_pc, gt_box, gt_mask, traj):
    diou = distance_iou(pred_box, gt_box)
    mask_iou = intersection_over_union(pred_mask, gt_mask)
    absrel = mean(abs(pred_depth - gt_depth) / gt_depth)
    delta125 = ratio(pred_depth / gt_depth < 1.25)
    cd = chamfer_distance(pred_pc, gt_pc) / scene_diagonal
    centroid = l2_distance(mean(pred_pc), mean(gt_pc))
    ra_dino = dino_similarity(pred_img, gt_img) * exp(-1.0 * e_parallel - 0.8 * e_orthogonal)
    deqa = perceptual_quality_score(pred_img)
    phys_vlm = vlm_physical_plausibility_score(pred_img, traj)
    # All metrics are linearly normalized to [0,100] per paper convention
    return {k: v * 100 for k, v in locals().items()}
```

## Common pitfalls

- All reported metrics are linearly normalized to [0, 100] for the final table, which differs from standard raw metric ranges.
- Chamfer distance is computed on point clouds normalized by the valid-scene diagonal, not raw coordinates.
- RA-DINO modifies standard DINO similarity by explicitly penalizing parallel and orthogonal relocation vector errors.

## Evidence (verbatim from paper)

> Data. Since there is no widely accepted benchmark for this task, we build an evaluation set with 200 image pairs and about 320 individual objects. It covers diverse scenes, object categories, and object scales. Half of the pairs contain a single manipulated object, and the other half contain multiple manipulated objects. Each pair includes depth annotations and object-level labels. Metrics. We report metrics from five aspects. ... All metrics are linearly normalized to $[0,100]$. Methods are sorted by Chamfer distance in descending order.

## Citation

```bibtex
@misc{xu2026phyedit,
  title={PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing},
  author={Xu et al. (2026)},
  year={2026},
  note={arXiv:2604.07230}
}
```

- arXiv: 2604.07230