2600 Data Driven Optimize Output Analysis E58bea19

# Guide to Analyzing Data-Driven Optimize Output

## 1. Prerequisites: GCS Authentication

Before using the Python tools described in this guide, you must authenticate
your local environment with Google Cloud. The tools need to access the output
files stored in a GCS bucket, which requires proper permissions.

## 2. Introduction

After a Data-Driven Optimize (Prompt Optimizer) job completes, it writes a
number of files to the specified output path. This can be a Google Cloud
Storage (GCS) bucket or a local directory. This document explains what these
files are and how to use the provided tools to analyze them.

## 3. Data-Driven Optimize Output Files

The output directory contains several JSON files that capture the results of
the optimization.

-   **`optimized_results.json`**: Contains the single best prompt template found.

The entries in the following pairs of files have corresponding orders, and
their contents can be zipped together for detailed analysis:

-   **Training Results During Optimization Process**:
    -   **`templates.json`**: Contains all templates evaluated during the
        optimization process, along with their summary metrics.
    -   **`eval_results.json`**: Contains detailed, per-row training results on
        minibatches during the optimization process. This file is for manual,
        in-depth analysis and is not read by the analysis tools.
-   **Initial and Best Templates on Full Test Set**:
    -   **`test_templates.json`**: Contains your original prompt and the best
        prompt, along with their summary metrics on the full test dataset. The
        best prompt is selected based on the metric from training only.
    -   **`test_eval_results.json`**: Contains the detailed, per-row evaluation
        scores for the original and best prompts on the full test dataset.
        This file is for manual, in-depth analysis and is not read by the
        analysis tools.
-   **Eval of All Candidates on Full Test Set**:
    -   **`all_candidates_full_test_templates.json`**: A list of *all* prompt
        candidates, along with their summary metrics on the full test set.
    -   **`all_candidates_full_test_eval_results.json`**: Contains detailed,
        per-row evaluation results for each candidate prompt on the full test
        set. This file is for manual, in-depth analysis and is not read by the
        analysis tools.
-   **Eval of All Candidates on Full Training Set**:
    -   **`all_candidates_full_train_templates.json`**: A list of *all* prompt
        candidates, along with their summary metrics on the full training set.
    -   **`all_candidates_full_train_eval_results.json`**: Contains detailed,
        per-row evaluation results for each candidate prompt on the full
        training set. This file is for manual, in-depth analysis and is not
        read by the analysis tools.

### Controlling Output File Generation

Not all output files are generated by default. Their creation depends on your
job's configuration. This is important because the analysis tools are designed
to handle missing files gracefully, but you may want to enable certain outputs
for a more complete analysis.

Here are the key flags that control which files are generated:

-   **Providing Test Data (`test_input_data_path`)**: This parameter is
    **optional but highly recommended**. When you provide a test dataset, the
    `test_templates.json` and `test_eval_results.json` files are generated,
    which provide a final comparison between your original prompt and the
    best-optimized prompt. Without a test set, you cannot accurately measure
    the generalization performance of your prompts.

-   **`eval_all_candidates_on_full_test`**: This is an **optional** boolean
    flag. When set to `True` (and a test set is provided), the job will generate
    `all_candidates_full_test_templates.json` and
    `all_candidates_full_test_eval_results.json`. These files allow for a
    deep-dive analysis into the performance of *every* candidate on the test
    set. **Warning:** Enabling this flag can be computationally expensive and
    significantly increase the job's runtime.

-   **`eval_all_candidates_on_full_train`**: This is an **optional** boolean
    flag. When set to `True`, the job will generate
    `all_candidates_full_train_templates.json` and
    `all_candidates_full_train_eval_results.json`. These enable a detailed
    comparison of performance on the full training set versus the test set, and
    against batch training results, which is crucial for identifying
    overfitting. **Warning:** Enabling this flag can be computationally
    expensive and significantly increase the job's runtime.

## 4. Data File Structure Details

**Key Fields:**

-   In `_templates.json` files (e.g., `all_candidates_full_train_templates.json`,
    `test_templates.json`):
    -   **`step`**: The optimization step where this candidate was generated or
        selected.
    -   **`metrics`**: This object contains aggregated evaluation results
        (e.g., `ROUGE_L/mean` or `exact_match/mean`). The specific metric name
        will vary depending on the `eval_metric` configured. It also contains
        the "uid" which is an unique identifier consistent across all result
        files.
    -   **`prompt`**: The prompt string itself.

-   In `_eval_results.json` files (e.g., `all_candidates_full_train_eval_results.json`,
    `test_eval_results.json`):
    -   **`summary_results`**: This object contains the aggregated metrics for
        the entire evaluation.
    -   **`metrics_table`**: This object contains the per-row evaluation
        results, useful for detailed debugging.

## 5. Generating an Analysis Report

The recommended way to analyze your results is to use a multi-step process
involving the tools provided and your reasoning capabilities:

1.  **Analyze Results:** First, call
    `analyze_data_driven_optimize_results(output_path, analysis_data_path="path/to/analysis_data.json")`
    to process the raw output files. Providing `analysis_data_path` is highly
    recommended as it saves the results into three separate files to avoid
    system size limits:
    -   `analysis_data.json`: Core metadata (config, comparison, best prompt).
    -   `analysis_data_metrics.json`: Detailed metrics for all candidates
        (no prompt text).
    -   `analysis_data_prompts.json`: Mapping of top candidates' keys to full
        prompt texts.
    This tool returns a **minimal summary object** with the paths to the saved
    files and the best prompt score.
2.  **Formulate Suggestions:** You should then analyze the `analysis_data`
    summary. Based on the guidelines in the **Data-Driven Optimize Tuning Guide**,
    formulate suggestions for the next run. This should result in a
    `suggested_config_data` dictionary, containing:
    -   `"suggested_config"`: A dictionary of configuration modifications.
    -   `"rationale"`: A string explaining the reasoning behind the
        suggestions.
3.  **Generate HTML Report:** Finally, call `generate_html_report()` passing
    either the `analysis_data` object or the `analysis_data_path` obtained in
    Step 1, along with the `suggested_config_data`. Passing `analysis_data_path`
    is the most efficient way to handle large analysis results as it
    automatically re-joins the metadata, metrics, and prompts from the three
    split files.

You can then open the generated HTML file in any web browser to view the full
analysis. For a deeper understanding of how to interpret these results and
iterate, please refer to the Data-Driven Optimize Tuning Guide.