2601 Data Driven Optimize Tuning Guide 402eab49

# Guide to Tuning Data-Driven Optimize Jobs

## 1. Introduction

After reviewing the initial results from a Data-Driven Optimize job, the next
step is to decide how to improve them in the next run. This guide provides a
framework for interpreting the results to diagnose common issues and suggests
concrete changes to your configuration for the next iteration.

## 2. Core Diagnostic Questions

The analysis of a Data-Driven Optimize run can be distilled into few key
questions.

### Question 0: What do we see about the data?

It's crucial to understand the data presented in the
`analyze_data_driven_optimize_results` tool output. This tool processes the raw
output files and provides a structured JSON object containing all the necessary
information for analysis.

1.  **Examine the Analysis Output:** Inspect the JSON object returned by
    `analyze_data_driven_optimize_results`. Key top-level keys include:
    -   `config`: The configuration used for the Data-Driven Optimize job
        (summarized if saving to file).
    -   `best_prompt`: The prompt with the highest score based on the
        `metric_source` (present in the metadata file).
    -   `comparison`: Metrics comparing the initial prompt and the best prompt
        on the training batches and full test set (if available).
    -   `all_metrics_ranked`: A list of all candidates and their scores across
        different evaluations. **Note: Full prompt text is excluded from this
        list to save space and is stored separately. When saving to a file,
        this list is omitted from the chat summary to avoid payload limits.**
        This list contains the data needed to analyze learning curves and is
        available in the `*_metrics.json` file.
    -   `metric_name`: The name of the metric being optimized.
    -   `metric_source`: Indicates the data source used for ranking in
        `all_metrics_ranked`.

2.  **Identify Available Datasets and Metrics:** The `all_metrics_ranked` list
    is the most important part for this. Each element in this list is a
    dictionary representing a prompt candidate, and contains scores from *all*
    available evaluation configurations. The presence of certain keys within
    these dictionaries indicates which datasets were evaluated:
    -   **`score_ucb_train`**: Always present. Represents the score on the
        training mini-batches.
    -   **`score_full_train`**: Present if `eval_all_candidates_on_full_train`
        was true in the config.
    -   **`score_full_test`**: Present if `eval_all_candidates_on_full_test`
        was true in the config.

    The `comparison` object will also reflect metrics for the initial and best
    prompts, typically on the test set data if configured. Understanding the
    distinctions between these datasets is vital for diagnosing issues like
    overfitting or noisy evaluations.

3.  **Assess Data Noise:** To analyze trends and noise, you need to process the
    `all_metrics_ranked` list to extract scores for each `step`. For example,
    examine the variance or trends of `score_ucb_train`, `score_full_train`, and
    `score_full_test`.
    -   **Noisy Data:** Significant fluctuations (e.g., large ups and downs) in
        training scores across steps suggest noise in the evaluations.
    -   **Stable Data:** Relatively smooth trends in scores suggest more
        reliable evaluations.

### Question 1: Is the optimization process actually learning?

First, we need to determine if the optimizer is successfully finding better
prompts on the **training data**. To do this, analyze the training score values
within the `all_metrics_ranked` list. You should look at the trend of the
maximum training score at each step.

-   **What to look for:** A generally upward trend in the training score across
    the optimization steps.
-   **Good sign:** The score is increasing from one step to the next. This
    means the algorithm is generating better prompts and successfully
    identifying them.
-   **Bad sign:** The training score is flat, erratic, or decreasing. This
    suggests a problem with the optimization process itself or noise in the
    metric.

### Question 2: Is the learning generalizing to unseen data?

If the model is learning on the training data, we then need to see if that
learning translates to better performance on the **full test data**. This tells
us if we are overfitting.

-   **What to look for:** Compare the trend of the training score with the test
    score at each step within the `all_metrics_ranked` data. This requires test
    scores to be available.
-   **Good sign:** Both the train score and the test score are increasing. This
    is the ideal scenario, indicating that the prompts are getting better and
    the improvements are generalizing.
-   **Bad sign (Overfitting):** The training score is increasing, but the test
    score is flat or decreasing. This is a classic sign of overfitting. The
    optimizer is finding prompts that are too specific to your training
    examples and do not perform well on new, unseen examples.
-   **Other Gaps:** Also, pay attention to the gap between mini-batch training
    scores and full training scores (if available). A large or inconsistent gap
    here might suggest that the mini-batches are not representative of the full
    training set, potentially due to a small `batch_size`.

### Question 3: Is the starting metric too high and already saturated?

## 3. Tunable Parameters

**Important**: The parameters listed in this section are the primary and
**only** tuning knobs that should be modified for Data-Driven Optimize jobs.
Other parameters are not intended for user modification and can lead to
unexpected results.

The following parameters are the knobs that can be adjusted to tune the
Data-Driven Optimize optimization process:
-   `num_steps`
-   `batch_size`
-   `population_size`
-   `num_mutants_per_candidate`
-   `num_ucb_steps`

### Impact of Tuning Knobs

-   **`num_steps`**: The total number of optimization steps to run. More steps
    allow for more exploration but increase runtime.
-   **`batch_size`**: The number of data points used to evaluate a candidate
    prompt in each mini step. Larger batches provide more stable score
    estimates but increase the computational cost of each step. A common batch
    size is between 20-50.
-   **`population_size`**: Controls the number of elite candidates preserved at
    each iteration. A larger size maintains more diversity, potentially
    avoiding local optima but slowing down convergence.
-   **`num_mutants_per_candidate`**: Determines the exploration breadth around
    each elite candidate. More mutants increase the chances of finding
    improvements but require more evaluations.
-   **`num_ucb_steps`**: Controls how many times candidates are selected and
    evaluated *within* an iteration. More steps lead to more accurate
    estimates of each candidate's performance before selection, but increases
    the computational cost per iteration.

## 4. Common Scenarios and Next Steps

**General Guidance:**

-   **Prioritize Addressing Noise:** If the metrics appear noisy, address this
    first by increasing `batch_size` and/or `num_ucb_steps`. It is difficult to
    diagnose other issues like overfitting if the underlying metrics are not
    stable.
-   If the training score is not improving, increasing `num_ucb_steps` or
    `batch_size` can help get more stable estimates for candidate selection.
-   To increase exploration, you can increase `population_size` or
    `num_mutants_per_candidate`.
-   **Resource Consideration:** The total number of model inference calls per
    iteration is determined by a combination of the number of candidates in the
    pool, the number of selection steps (`num_ucb_steps`), and the `batch_size`.
    Balancing these parameters is crucial to achieve the best quality with
    highest efficiency. When suggesting changes, always consider the trade-off
    between potential quality gains and the increased computational cost.

Based on the answers to the questions above, here are some common scenarios and
suggested actions. It's best to first check for and address noisy metrics
before diving into other potential issues.

### Scenario 1: Metrics are Noisy or Train Score is Flat/Erratic

**Diagnosis:** Scores fluctuate significantly across steps, or the training
score is flat or erratic. This makes it difficult to determine if changes in
scores are due to better prompts or random chance.

**Suggestions:**
-   **Increase `batch_size`:** This is often the most effective way to reduce
    noise, as it provides a more stable estimate of prompt performance on a
    larger sample of data.
-   **Increase `num_ucb_steps`:** More selection steps allow for more
    evaluations of each candidate, which can help in getting more reliable
    score estimates before the next generation.
-   **Improve Data Quality:** Examine the data for inconsistencies or errors. A
    cleaner dataset leads to a more stable metric.
-   **Run on the full dataset (for small datasets):** If your training dataset
    is relatively small, the noise from sampling can be high. Consider
    configuring the job to evaluate on the full training dataset at each step.

### Scenario 2: Both Train and Test scores are improving.

**Diagnosis:** This is the ideal outcome. The process is working as expected.

**Suggestion:**
-   You can stop if you are satisfied with the performance.
-   To potentially achieve even better results, you can try another run using
    the best prompt from this job as the new starting point.

### Scenario 3: Train score improves, but Test score is flat or decreasing.

**Diagnosis:** Overfitting. The prompts are becoming too tailored to the
training data.

**Suggestions:**
-   **Add more data:** The most effective way to combat overfitting is to
    increase the number of diverse, high-quality examples in your training set.
-   **Enable full test evaluation:** For a more granular view of overfitting,
    you can evaluate every candidate prompt on the full test set. This allows
    for a detailed analysis of how each candidate generalizes.
-   **Enable full training evaluation:** For a detailed comparison between
    training and test performance, you can evaluate all candidate prompts
    against the full training set. This identifies performance gaps between
    seen and unseen data.

## 5. Advanced Tuning Lessons

This section provides a more structured approach to tuning, based on the
`analyze_data_driven_optimize_results` output.

### I. Key Signals to Examine

Before making changes, diagnose the current state of optimization by looking at
these specific signal combinations:

1.  **The "Best" Step Location**
    -   **What to look for:** At what step do the highest scores occur?
    -   **Why:** This indicates if the process is "undertrained" (peaked at the
        end) or "saturated" (peaked early).
2.  **Metric Stability**
    -   **What to look for:** How much do training scores fluctuate?
    -   **Why:** High volatility usually indicates that the sample size is too
        small to get a stable estimate of prompt quality.
3.  **Generalization Gaps**
    -   **What to look for:** How do the trends of training and test scores
        compare?
    -   **Why:** This separates healthy learning from overfitting.

### II. Tuning Guidelines & Heuristics

1.  **The Hierarchy of Levers**
    When results are unstable or stalling, apply fixes in this order:
    -   **Increase `batch_size` (Primary):** This is the most effective lever
        for stabilizing metrics and providing a smoother optimization gradient.
    -   **Increase `num_ucb_steps` (Secondary):** Use this to further refine
        candidate selection.
    -   **Adjust `num_steps` (Tertiary):** Change this only after the metric is
        stable.

### III. Budget and Resource Management

Tuning parameters have a direct impact on the number of model inference calls,
and thus the overall cost and time required for optimization.

1.  **Resource-Neutral Tuning**
    If you have a fixed budget, you must trade off one parameter for another:
    -   **To increase exploration without increasing cost:** If you increase
        the number of mutant variants, you should decrease the population size
        or batch size proportionally.
    -   **To increase stability without increasing cost:** If you increase the
        batch size, you must decrease the number of tracked candidates or
        selection steps.

2.  **Cost Efficiency Heuristic**
    -   **Increasing the number of steps** is the most direct way to spend
        more budget. It doesn't change the complexity of each step, just the
        duration.
    -   **Increasing the population size** is the most expensive way to spend
        budget, as it increases the number of candidates that must be tracked
        and evaluated in every subsequent step.

## 6. New suggested config

Name the new config file base on the original config name. Add a suffix index
and a name to indicate the change. Only modify the Tuning Knobs and Inherit the
rest of the parameters from the original config.