Gibson Env Eval

---
name: gibson-env-eval
description: Evaluates the geometric complexity, scene diversity, and sim-to-real transfer capability of the Gibson virtual environment. It benchmarks neural rendering pipelines and embodied agents on tasks like depth estimation, scene classification, and navigation. Use when the user wants to benchmark on Gibson, or asks about evaluating this task. Reports Real-World Transfer Error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1808.10654
  bibtex_key: xia2018gibson
  confidence: high
---

# gibson-env-eval

> Gibson Env: Real-World Perception for Embodied Agents — Xia et al. (2018) (arXiv:1808.10654, 2018)

## What this evaluates

Evaluates the geometric complexity, scene diversity, and sim-to-real transfer capability of the Gibson virtual environment. It benchmarks neural rendering pipelines and embodied agents on tasks like depth estimation, scene classification, and navigation.

## Datasets

- **Gibson** — total 572; splits: full (572)

## Metrics

- `Real-World Transfer Error` **(primary)** — range: meters
  - Average depth estimation error in meters when a network trained on synthetic database images is tested on real-world 2D-3D-S images.
- `Depth Estimation Error` — range: meters
  - Mean absolute difference between predicted and ground truth depth maps, measured in meters.
- `Scene Class Accuracy` — range: [0, 1]
  - Top-1 classification accuracy on scene categories using a pretrained network on 10k random images per database.
- `SSIM` — range: [0, 1]
  - Structural Similarity Index Measure quantifying perceptual image similarity between rendered and real images.
- `L1` — range: [0, 1]
  - Mean absolute pixel-wise difference between rendered and real images.
- `MMD` — range: other
  - Maximum Mean Discrepancy computed on VGG16 features to measure distributional distance between synthetic and real domains.
- `CORAL` — range: other
  - Correlation Alignment distance computed on VGG16 features to measure distributional distance between synthetic and real domains.
- `Average Reward` — range: other
  - Mean cumulative reward over evaluation iterations for navigation tasks, incorporating distance decrease and collision penalties.

## Input / output format

**Input**: RGB-D panorama images (1024x2048 or cropped 256x256) and agent state (position, orientation, target relative position).

**Output**: Predicted depth maps, scene class labels, action logits (forward/backward/left/right or continuous torques), and rendered images.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    depth_err = np.mean(np.abs(preds['depth'] - gold['depth']))
    class_acc = np.mean(preds['class'] == gold['class'])
    ssim_val = ssim(preds['img'], gold['img'])
    l1_val = np.mean(np.abs(preds['img'] - gold['img']))
    mmd_val = max_mean_discrepancy(preds['features'], gold['features'])
    coral_val = correlation_alignment(preds['features'], gold['features'])
    avg_reward = np.mean(preds['rewards'])
    return {'depth_err': depth_err, 'class_acc': class_acc, 'ssim': ssim_val, 'l1': l1_val, 'mmd': mmd_val, 'coral': coral_val, 'avg_reward': avg_reward}
```

## Common pitfalls

- Domain gap between synthetic and real images can cause depth estimation errors to inflate significantly if the Goggles domain-adaptive network is not applied.
- Navigation complexity metric depends on A* pathfinding on reconstructed meshes, which may vary with mesh quality and post-processing steps.
- Scene diversity entropy calculation relies on a fixed pretrained classifier, making it sensitive to the classifier's inherent bias and not a pure measure of environment variety.

## Evidence (verbatim from paper)

> Real-World Transfer Error: We train a neural network for depth estimation using the images of each database and test them on real images of 2D-3D-S dataset [10]. Training images of SUNCG and Matterport3D are rendered using MINOS [76] and our dataset is rendered using Gibson's engine. The training set of each database is 20k random RGB-depth image pairs with 90∘ field of view. The reported value is average depth estimation error in meters.

## Citation

```bibtex
@misc{xia2018gibson,
  title={Gibson Env: Real-World Perception for Embodied Agents},
  author={Xia et al. (2018)},
  year={2018},
  note={arXiv:1808.10654}
}
```

- arXiv: 1808.10654