Libero Cf Eval

---
name: libero-cf-eval
description: Evaluates whether Vision-Language-Action (VLA) models can follow counterfactual language instructions in robotic manipulation tasks. It specifically probes for 'vision shortcuts' where models default to well-learned visual behaviors instead of adhering to the given text commands. Use when the user wants to benchmark on LIBERO-CF, or asks about evaluating this task. Reports grounding rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.17659
  bibtex_key: fang2026when
  confidence: high
---

# libero-cf-eval

> When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs — Fang et al. (2026) (arXiv:2602.17659, 2026)

## What this evaluates

Evaluates whether Vision-Language-Action (VLA) models can follow counterfactual language instructions in robotic manipulation tasks. It specifically probes for 'vision shortcuts' where models default to well-learned visual behaviors instead of adhering to the given text commands.

## Datasets

- **LIBERO-CF** — total 50; splits: test (50)

## Metrics

- `grounding rate` **(primary)** — range: percent
  - Measures whether the gripper makes contact with the target object specified in the instruction, regardless of whether the full task is completed.
- `success rate` — range: percent
  - Measures whether the robot successfully completes the instructed task from start to finish.

## Input / output format

**Input**: RGB images of the robot scene and natural language task instructions.

**Output**: Robot action commands (e.g., gripper positions, joint velocities) executed over a trajectory.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    grounding_hits = 0
    success_hits = 0
    for pred, gold_inst in zip(predictions, gold):
        if pred.contact_object == gold_inst.target_object:
            grounding_hits += 1
        if pred.task_completed == gold_inst.completed:
            success_hits += 1
    return {
        'grounding_rate': (grounding_hits / len(predictions)) * 100,
        'success_rate': (success_hits / len(predictions)) * 100
    }
```

## Common pitfalls

- Models often default to manipulating training-task objects even when instructed otherwise, due to strong visual priors.
- Language-only inference causes performance to collapse to near zero, masking the model's actual visual capabilities.
- Grounding rate can be significantly higher than success rate, as contacting the target object does not guarantee task completion.

## Evidence (verbatim from paper)

> To quantitatively evaluate language following behaviors, we define a grounding rate (“grounding”), which measures whether the gripper makes contact with the target object specified in the instruction, regardless of task completion. In addition, we report the success rate (“success”), which measures whether the robot completes the task, providing a stricter measure of task execution.

## Citation

```bibtex
@misc{fang2026when,
  title={When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs},
  author={Fang et al. (2026)},
  year={2026},
  note={arXiv:2602.17659}
}
```

- arXiv: 2602.17659