Interndata A1 Real Eval

---
name: interndata-a1-real-eval
description: Evaluates the zero-shot sim-to-real transfer and generalization of a Vision-Language-Action (VLA) policy on diverse real-world and simulated manipulation tasks. It probes fundamental pick-and-place, articulated object manipulation, human-robot interaction, and long-horizon task composition capabilities. Use when the user wants to benchmark on InternData-A1 Real-World & Sim-to-Real Benchmarks, or asks about evaluating this task. Reports average success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.16651
  bibtex_key: tian2025interndataa1
  confidence: high
---

# interndata-a1-real-eval

> InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policy — Yang Tian et al. (2025) (arXiv:2511.16651, 2025)

## What this evaluates

Evaluates the zero-shot sim-to-real transfer and generalization of a Vision-Language-Action (VLA) policy on diverse real-world and simulated manipulation tasks. It probes fundamental pick-and-place, articulated object manipulation, human-robot interaction, and long-horizon task composition capabilities.

## Datasets

- **InternData-A1 Real-World & Sim-to-Real Benchmarks** — total ?; splits: test (-1)

## Metrics

- `average success rate` **(primary)** — range: percent
  - Calculated as the number of successful rollouts divided by the total number of rollouts (30 per task). A trial is successful only if all task-specific physical and procedural criteria are met.

## Input / output format

**Input**: Multi-view visual observations (head and wrist cameras) and natural language instructions describing the manipulation task.

**Output**: Robot action commands (joint positions/velocities or end-effector poses) for dual-arm manipulation.

## Scoring recipe

```python
successes = 0
for rollout in range(30):
    if check_task_success(task_id, rollout_state):
        successes += 1
return (successes / 30) * 100
```

## Common pitfalls

- Success criteria are strictly defined per task (e.g., 'fully and correctly zipped'), so minor deviations count as failures.
- High stochasticity in real-world physics and perception requires averaging over 30 rollouts per task to obtain stable metrics.
- Sim-to-real transfer performance may vary significantly across different robot embodiments (Genie-1, ARX Lift-2, ARX AC One) due to hardware-specific control latencies.

## Evidence (verbatim from paper)

> For each task, we define 15 evaluation settings, and to reduce stochasticity, we run two trials per setting. In total, each task is evaluated with 30 rollouts, and we report the average success rate.

## Citation

```bibtex
@misc{tian2025interndataa1,
  title={InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policy},
  author={Yang Tian et al. (2025)},
  year={2025},
  note={arXiv:2511.16651}
}
```

- arXiv: 2511.16651