Embodiedcomp Eval

---
name: embodiedcomp-eval
description: Evaluates how image compression codecs impact the performance of Vision-Language-Action (VLA) models in closed-loop robotic manipulation tasks under ultra-low bitrates. It measures whether compressed visual inputs cause task failure or require excessive inference steps, highlighting the disconnect between traditional visual fidelity metrics and embodied AI operational requirements. Use when the user wants to benchmark on EmbodiedComp, or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.11612
  bibtex_key: li2025embodied
  confidence: high
---

# embodiedcomp-eval

> Embodied Image Compression — Li et al. (2025) (arXiv:2512.11612, 2025)

## What this evaluates

Evaluates how image compression codecs impact the performance of Vision-Language-Action (VLA) models in closed-loop robotic manipulation tasks under ultra-low bitrates. It measures whether compressed visual inputs cause task failure or require excessive inference steps, highlighting the disconnect between traditional visual fidelity metrics and embodied AI operational requirements.

## Datasets

- **EmbodiedComp** — total ?; splits: train (2000), test (100)

## Metrics

- `Success Rate (SR)` **(primary)** — range: [0, 1]
  - The fraction of test scenes in which the VLA successfully executes the commanded primitive action (pick, push, or press) within the maximum iteration step budget.
- `Step` — range: other
  - The number of VLA inference iterations required to complete the task or reach the maximum step budget. Exceeding 150 steps is treated as task failure.

## Input / output format

**Input**: Compressed/distorted 256x256 images rendered from a Robosuite/MuJoCo simulation (third-person or first-person gripper view), paired with natural language commands referring to a single main object.

**Output**: Discrete action commands (pick, push, or press) predicted by the VLA, which are executed in the simulator to update the environment state.

## Scoring recipe

```python
success_count = 0
steps_list = []
for scene in test_scenes:
    state = scene.initial_state
    for step in range(1, max_steps + 1):
        img = render(state)
        img_compressed = compress(img, target_bpp)
        action = vla.predict(img_compressed, command)
        state = step_simulation(state, action)
        if is_success(state, command):
            success_count += 1
            steps_list.append(step)
            break
    else:
        steps_list.append(max_steps)
sr = success_count / len(test_scenes)
avg_steps = sum(steps_list) / len(test_scenes)
```

## Common pitfalls

- Using traditional image fidelity metrics (e.g., PSNR, SSIM) instead of task-based metrics, as the paper demonstrates a fundamental disconnect between CV-based fidelity and embodied agent performance.
- Evaluating zero-shot VLAs directly, which the protocol explicitly forbids; models must be fine-tuned on the train split first to isolate compression distortion from policy limitations.
- Misinterpreting the Step metric: exceeding 150 iterations is classified as failure, not merely a high cost, due to the positive-feedback degradation regime.

## Evidence (verbatim from paper)

> Consequently, EmbodiedComp adopts only two indicators: (i) Success Rate (SR): the fraction of scenes in which the command is ultimately satisfied. (ii) Step: the number of VLA iterations to reach success or exhaust the budget.

## Citation

```bibtex
@misc{li2025embodied,
  title={Embodied Image Compression},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2512.11612}
}
```

- arXiv: 2512.11612