979 Skill B9e49017

---
name: synth-api
description: Use the Synth AI API end-to-end (SDK + HTTP) for eval + GEPA
---

# Synth API (SDK + HTTP, end-to-end)

This skill explains how to run Synth end-to-end with:
- a **Local API task app** exposed via **Cloudflare tunnel**
- **GEPA prompt optimization**
- **Eval jobs** on held-out seeds

Reference demo: `demos/gepa_banking77/gepa_banking77_prompt_optimization.ipynb`

## Required env

- `SYNTH_API_KEY`: your API key (or mint a demo key if using a demo workflow)
- `SYNTH_BACKEND_URL` (optional): backend base URL, default `https://api.usesynth.ai`

## Auth + API keys

Synth uses two keys:

- `SYNTH_API_KEY` authenticates your SDK/CLI calls to the Synth backend.
- `ENVIRONMENT_API_KEY` authenticates backend-to-task-app requests (sent as `X-API-Key` or `Authorization: Bearer ...`).

### Mint a demo Synth API key (optional)

Demo keys are short‑lived (default 4 hours) and are great for notebooks or quick starts.

```python
import os

from synth_ai.core.utils.env import mint_demo_api_key

SYNTH_API_BASE = os.environ.get("SYNTH_BACKEND_URL", "https://api.usesynth.ai")
SYNTH_API_KEY = os.environ.get("SYNTH_API_KEY") or mint_demo_api_key(SYNTH_API_BASE)
os.environ["SYNTH_API_KEY"] = SYNTH_API_KEY
```

### Mint + upload an Environment API key

Your task app should use the same `ENVIRONMENT_API_KEY` that the backend stores for your org.
The helper below generates a key locally and uploads it to the backend using your `SYNTH_API_KEY`.

```python
import os

from synth_ai.sdk.localapi.auth import mint_environment_api_key, setup_environment_api_key

SYNTH_API_BASE = os.environ.get("SYNTH_BACKEND_URL", "https://api.usesynth.ai")
SYNTH_API_KEY = os.environ["SYNTH_API_KEY"]

ENVIRONMENT_API_KEY = mint_environment_api_key()
os.environ["ENVIRONMENT_API_KEY"] = ENVIRONMENT_API_KEY

setup_environment_api_key(SYNTH_API_BASE, SYNTH_API_KEY, token=ENVIRONMENT_API_KEY)
```

## Core concepts

- **Local API**: Your task app runs locally and exposes `/rollout` + `/task_info`.
- **Tunnel**: Cloudflare Quick Tunnel makes the local app reachable by Synth.
- **GEPA**: Prompt optimizer that mutates prompts to maximize reward.
- **Eval jobs**: Formal evaluation on held-out seeds after optimization.

## Quick SDK health check

```python
import os
import asyncio

from synth_ai.sdk.jobs import JobsClient


async def main() -> None:
    async with JobsClient(
        base_url=os.environ.get("SYNTH_BACKEND_URL", "https://api.usesynth.ai"),
        api_key=os.environ["SYNTH_API_KEY"],
    ) as client:
        files = await client.files.list(limit=5)
        print(files)


if __name__ == "__main__":
    asyncio.run(main())
```

## 1) Define a Local API task app

Minimum Local API shape:
- `provide_taskset_description()`
- `provide_task_instances(seeds)`
- `rollout(request) -> RolloutResponse`

```python
from synth_ai.sdk.localapi import LocalAPIConfig, create_local_api
from synth_ai.sdk.localapi._impl.contracts import RolloutMetrics, RolloutRequest, RolloutResponse, TaskInfo


def create_banking77_local_api(system_prompt: str):
    async def run_rollout(request: RolloutRequest, fastapi_request) -> RolloutResponse:
        # Use your own task logic here; return a reward in [0, 1].
        reward = 1.0
        return RolloutResponse(
            trace_correlation_id=request.trace_correlation_id,
            reward_info=RolloutMetrics(outcome_reward=reward),
            trace=None,
        )

    def provide_taskset_description():
        return {"splits": ["train", "test"], "sizes": {"train": 1000, "test": 1000}}

    def provide_task_instances(seeds):
        for seed in seeds:
            yield TaskInfo(
                task={"id": "banking77", "name": "Banking77 Intent Classification"},
                dataset={"id": "banking77", "split": "train", "index": seed},
                inference={"tool": "banking77_classify"},
                limits={"max_turns": 1},
                task_metadata={"seed": seed},
            )

    return create_local_api(
        LocalAPIConfig(
            app_id="banking77",
            name="Banking77 Intent Classification",
            description="Classify customer queries into intents.",
            provide_taskset_description=provide_taskset_description,
            provide_task_instances=provide_task_instances,
            rollout=run_rollout,
            cors_origins=["*"],
        )
    )
```

## 2) Expose the Local API with a Cloudflare tunnel

Use the built‑in tunnel helper to auto‑start the server and provision a URL.
The helper spins up your local server, creates a public `trycloudflare.com` URL,
and forwards requests from the public URL to your local port. Keep the process
running while Synth calls your task app.

```python
from synth_ai.core.tunnels import TunnelBackend, TunneledLocalAPI

app = create_banking77_local_api("baseline prompt")
baseline_tunnel = await TunneledLocalAPI.create_for_app(
    app=app,
    local_port=None,  # auto-select
    backend=TunnelBackend.CloudflareQuickTunnel,
    progress=True,
)
LOCAL_API_URL = baseline_tunnel.url
print("Local API URL:", LOCAL_API_URL)
```

## 3) Run GEPA (prompt optimization)

GEPA mutates prompt candidates and evaluates them via rollouts. Use a GEPA config body
or a config file. Example config body:

```python
from synth_ai.sdk.optimization.internal.prompt_learning import PromptLearningJob

config_body = {
    "prompt_learning": {
        "algorithm": "gepa",
        "task_app_url": LOCAL_API_URL,
        "env_name": "banking77",
        "initial_prompt": {
            "messages": [
                {"role": "system", "order": 0, "pattern": "Baseline system prompt"},
                {"role": "user", "order": 1, "pattern": "Customer Query: {query}\n\nAvailable Intents:\n{available_intents}"},
            ],
            "wildcards": {"query": "REQUIRED", "available_intents": "OPTIONAL"},
        },
        "policy": {
            "model": "gpt-4.1-nano",
            "provider": "openai",
            "inference_mode": "synth_hosted",
            "temperature": 0.0,
            "max_completion_tokens": 256,
        },
        "gepa": {
            "env_name": "banking77",
            "evaluation": {
                "seeds": list(range(50)),
                "validation_seeds": list(range(50, 60)),
            },
            "rollout": {"budget": 80, "max_concurrent": 8, "minibatch_size": 8},
            "mutation": {"rate": 0.3},
            "population": {"initial_size": 4, "num_generations": 3, "children_per_generation": 3},
            "archive": {"size": 5, "pareto_set_size": 10},
        },
    },
}

job = PromptLearningJob.from_dict(config_dict=config_body, skip_health_check=True)
job_id = job.submit()
result = job.poll_until_complete(timeout=3600.0, interval=3.0, progress=True)
print(result.status.value)
```

## 4) Run Eval jobs (held‑out seeds)

Eval jobs score a fixed set of held‑out seeds for a final report once optimization
finishes.

```python
from synth_ai.sdk.eval.job import EvalJob, EvalJobConfig

config = EvalJobConfig(
    local_api_url=LOCAL_API_URL,
    backend_url=os.environ.get("SYNTH_BACKEND_URL", "https://api.usesynth.ai"),
    api_key=os.environ["SYNTH_API_KEY"],
    env_name="banking77",
    seeds=list(range(100, 150)),
    policy_config={"model": "gpt-4.1-nano", "provider": "openai"},
    env_config={"split": "test"},
    concurrency=10,
)
job = EvalJob(config)
job.submit()
result = job.poll_until_complete(timeout=600.0, interval=2.0, progress=True)
print(result.status)
```

## 5) Retrieve optimized prompts

```python
from synth_ai.sdk.optimization.internal.learning.prompt_learning_client import PromptLearningClient

client = PromptLearningClient()
prompt_results = await client.get_prompts(job_id)
best_score = prompt_results.best_score
print("Best score:", best_score)
```

## HTTP example (raw)

```python
import os
import requests

base = os.environ.get("SYNTH_BACKEND_URL", "https://api.usesynth.ai")
resp = requests.get(
    f"{base}/api/health",
    headers={"Authorization": f"Bearer {os.environ['SYNTH_API_KEY']}"},
    timeout=30,
)
resp.raise_for_status()
print(resp.json())
```

## Troubleshooting checklist

- **Cloudflare tunnel**: Make sure `TunneledLocalAPI` returns a reachable URL; expect a `trycloudflare.com` URL.
- **Inference URL**: If using hosted inference, your model requests should point to `https://api.usesynth.ai/api/inference/v1`.
- **Auth**: Confirm `SYNTH_API_KEY` is set and valid.
- **Task app shape**: Ensure `/task_info` and `/rollout` return valid `RolloutResponse`.
- **Polling errors**: If a poll fails, re‑query `/api/prompt-learning/online/jobs/{job_id}` to confirm job status.