Ft Ncfm Eval

---
name: ft-ncfm-eval
description: Evaluates the performance and data efficiency of a Vision-Language-Action (VLA) model trained on a synthetically distilled coreset compared to models trained on full datasets. It probes long-horizon manipulation, multi-task skill acquisition, and generalization across spatial, object, goal, and temporal dimensions. Use when the user wants to benchmark on CALVIN, Meta-World, LIBERO, or asks about evaluating this task. Reports Success Rate (SR %), Average Task Completion Length (Avg. Len).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.16233
  bibtex_key: chen2025ftncfm
  confidence: high
---

# ft-ncfm-eval

> FT-NCFM: An Influence-Aware Data Distillation Framework for Efficient VLA Models — Chen et al. (2025) (arXiv:2511.16233, 2025)

## What this evaluates

Evaluates the performance and data efficiency of a Vision-Language-Action (VLA) model trained on a synthetically distilled coreset compared to models trained on full datasets. It probes long-horizon manipulation, multi-task skill acquisition, and generalization across spatial, object, goal, and temporal dimensions.

## Datasets

- **CALVIN** — total ?; splits: test (-1)
- **Meta-World** — total ?; splits: test (-1)
- **LIBERO** — total ?; splits: test (-1)

## Metrics

- `Success Rate (SR %)` **(primary)** — range: percent
  - Percentage of successful task executions out of total evaluation trials.
- `Average Task Completion Length (Avg. Len)` **(primary)** — range: other
  - Average number of steps or sub-tasks completed successfully in long-horizon evaluations.
- `Total Training Time (GPU-hours)` — range: other
  - Total GPU-hours required for the model to converge from random initialization, including any one-time preprocessing overhead.

## Input / output format

**Input**: RGB visual observations processed by a ViT-B/16 backbone and natural language instructions.

**Output**: Continuous or discrete action vectors for robot manipulation.

## Scoring recipe

```python
def compute_metrics(trials, total_gpu_seconds):
    success_count = sum(1 for t in trials if t.success)
    sr = (success_count / len(trials)) * 100
    
    total_len = sum(t.completed_steps for t in trials)
    avg_len = total_len / len(trials)
    
    time_hours = total_gpu_seconds / 3600
    
    return {'SR %': sr, 'Avg. Len': avg_len, 'Time (GPU-h)': time_hours}
```

## Common pitfalls

- Baseline training times for policy distillation methods (e.g., RLDG, DROC) include both teacher model training and distillation, which must be accounted for when comparing against the data-centric FT-NCFM overhead.
- The FT engine's preprocessing cost is a one-time investment that should be amortized over multiple training iterations, not treated as a per-epoch cost.
- Performance scaling is evaluated at specific data ratios (1%, 5%, 10%); results are not directly comparable across different compression rates without normalization.

## Evidence (verbatim from paper)

> We focus on two core metrics: Success Rate (SR %) or Average Task Completion Length (Avg. Len) to measure performance, and Total Training Time (GPU-hours) to measure efficiency.

## Citation

```bibtex
@misc{chen2025ftncfm,
  title={FT-NCFM: An Influence-Aware Data Distillation Framework for Efficient VLA Models},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2511.16233}
}
```

- arXiv: 2511.16233