Visual Instruction Data Builder

---
name: visual_instruction_data_builder
description: Build LLaVA-style visual instruction records from captions and boxes for mechanism-faithful recovery experiments.
---

# Visual Instruction Data Builder

Use this skill when a task needs to convert symbolic image evidence into a multimodal instruction-following record in the style of Visual Instruction Tuning. Do not use it to hallucinate image content; every generated field must be grounded in supplied captions or boxes.

## Inputs
- `image_id`: stable identifier for the image or proxy image.
- `captions`: one or more natural-language captions.
- `boxes`: object records with `label` and normalized `bbox` coordinates.
- `response_type`: one of `conversation`, `detail`, or `reasoning`.
- Optional `question` and `answer` strings.

## Outputs
A JSON-compatible record with `human_prompt`, `assistant_answer`, `symbolic_context`, `response_type`, `is_resource_derived`, and `resource_files` when available.

## Workflow
1. Validate that captions and boxes are non-empty.
2. Summarize captions and object labels as the symbolic visual context that a language-only teacher could read.
3. Create a response-type-specific user instruction while keeping the assistant answer in a separate field.
4. Attach provenance so recovery can prove the item was derived from allowed evidence.

## Validation
Run `python tests/test_visual_instruction_data_builder.py` or validate the full skill tree with `validate_skill_tree.py --run-tests`.

## Limitations
This skill does not call GPT-4 or inspect pixels. It creates deterministic proxy records that preserve the paper's caption/box-to-instruction mechanism.