Multi Dataset Unification Pipeline

---
name: multi-dataset-unification-pipeline
description: When the user requests to find, analyze, and merge multiple datasets from Hugging Face or similar repositories into a unified format. This skill handles the complete pipeline searching for datasets, examining their structure, parsing format specifications, converting diverse dataset formats (ToolACE, Glaive, XLAM, etc.) into a standardized schema, and merging them into a single output file. Key triggers include requests involving 'dataset unification', 'format conversion', 'merge datasets', 'Hugging Face datasets', 'tool-calling datasets', 'JSONL output', or when users provide a format specification document like 'unified_format.md'.
---
# Instructions

## 1. Understand the Request
- Identify the target datasets (names, IDs, or search queries).
- Locate and read any provided format specification (e.g., `unified_format.md`).
- Clarify the scope: number of entries per dataset, output file name/location.

## 2. Search and Validate Datasets
- Use `huggingface-dataset_search` to find each dataset.
- Use `huggingface-hub_repo_details` to get metadata and confirm availability.
- Note the dataset size and structure.

## 3. Analyze Dataset Structures
- Load a sample from each dataset using `datasets.load_dataset`.
- Examine the column names and a few sample entries.
- Identify the native format (e.g., ToolACE uses `system`/`conversations`, Glaive uses `conversations`/`tools`, XLAM uses `query`/`answers`/`tools`).

## 4. Convert to Unified Format
- **Always** refer to the provided `unified_format.md` (or equivalent) for the target schema.
- **Key Rules**:
    - `conversation_id`: Format as `{source_short_name}_{index}` (e.g., `toolace_0`).
    - `messages`: List of messages with `role` (`user`/`assistant`/`tool`), `content`, and optionally `tool_calls` or `tool_call_id`.
    - `tool_calls`: For assistant messages, include `id` (format `tool_call_{n}`), `name`, `arguments`.
    - `tools`: List of normalized tool definitions.
    - Remove system messages if they only contain tool instructions.
    - If assistant only made tool calls, set `content` to `null`.
    - Do not add tool return results or assistant replies not in the original data.
- Use the bundled `scripts/convert_and_merge.py` for reliable, deterministic conversion of ToolACE, Glaive, and XLAM formats.
- For new dataset formats, write a custom converter following the patterns in the script.

## 5. Merge and Output
- Limit entries per dataset as requested (default: first 500).
- Merge all converted entries into a single list.
- Write to a JSONL file in the workspace (default: `unified_tool_call.jsonl`).
- Verify the output: count entries, check file size, validate format against the specification.

## 6. Finalize
- Provide a summary: datasets found, entries converted, output location.
- Optionally, show a sample entry from each source for validation.
- Confirm the task is complete.