Featureid 3ds Eval

---
name: featureid-3ds-eval
description: Evaluates a derivative-free camera control policy's ability to align vision-language models in 3D multi-object scenes. It probes robustness to viewpoint changes and object occlusions using minimal demonstration data. Use when the user wants to benchmark on FeatureID-3DS, PartialView-3DS, or asks about evaluating this task. Reports prediction error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.24826
  bibtex_key: armitage2025video
  confidence: medium
---

# featureid-3ds-eval

> Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control — Armitage et al. (2025) (arXiv:2512.24826, 2025)

## What this evaluates

Evaluates a derivative-free camera control policy's ability to align vision-language models in 3D multi-object scenes. It probes robustness to viewpoint changes and object occlusions using minimal demonstration data.

## Datasets

- **FeatureID-3DS** — total ?; splits: test (-1)
- **PartialView-3DS** — total ?; splits: test (-1)

## Metrics

- `prediction error` **(primary)** — range: other
  - Computed over viewpoints where errors were marked in demonstrations and during the measurement round.

## Input / output format

**Input**: 3D scene coordinates (X, Y viewpoints, z-axis levels), demonstration data with marked errors, and noisy VLM outputs.

**Output**: Predicted camera actions for the correction round and updated demonstration data with system decisions.

## Scoring recipe

```python
# Pseudo-code based on section description
def evaluate(demonstrations, vlm_outputs, camera_state):
    # Measurement round
    system_decisions = update_demonstration_data(demonstrations, camera_state)
    coefficients = measure_coefficients(system_decisions, viewpoint_labels)
    
    # Correction round
    predicted_actions = predict_camera_actions(coefficients)
    
    # Scoring
    prediction_errors = compute_errors(predicted_actions, marked_viewpoints)
    return prediction_errors
```

## Common pitfalls

- Relies on external VLMs (llava-v1.5-13b, vicuna-13b-v1.5) without specifying exact prompt templates or inference parameters.
- Uses only 5% of scenes (n=3) for demonstrations, which may not capture full scene diversity.
- Hardware setup splits VLM inference (A100) and camera control (RTX 2080), but latency/throughput metrics are not reported in this section.

## Evidence (verbatim from paper)

> Methods receive prediction errors for viewpoints where an error was marked in demonstrations and during the measurement round. Demonstrations are set at 5% of scenes for each benchmark: n = 3 for feature identification (FeatureID-3DS) and object occlusion (PartialView-3DS).

## Citation

```bibtex
@misc{armitage2025video,
  title={Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control},
  author={Armitage et al. (2025)},
  year={2025},
  note={arXiv:2512.24826}
}
```

- arXiv: 2512.24826