Arize Trace

---
name: arize-trace
description: "INVOKE THIS SKILL when downloading or exporting Arize traces and spans. Covers exporting traces by ID, sessions by ID, and debugging LLM application issues using the ax CLI."
---

# Arize Trace Skill

## Concepts

- **Trace** = a tree of spans sharing a `context.trace_id`, rooted at a span with `parent_id = null`
- **Span** = a single operation (LLM call, tool call, retriever, chain, agent)
- **Session** = a group of traces sharing `attributes.session.id` (e.g., a multi-turn conversation)

Use `ax spans export` to download individual spans, or `ax traces export` to download complete traces (all spans belonging to matching traces).

**Resolving project for export:** The `PROJECT` positional argument accepts either a project name or a base64 project ID. When using a name, `--space-id` is required. If you hit limit errors or `401 Unauthorized` when using a project name, resolve it to a base64 ID: run `ax projects list --space-id SPACE_ID -l 100 -o json`, find the project by `name`, and use its `id` as `PROJECT`.

**Exploratory export rule:** When exporting spans or traces **without** a specific `--trace-id`, `--span-id`, or `--session-id` (i.e., browsing/exploring a project), always start with `-l 50` to pull a small sample first. Summarize what you find, then pull more data only if the user asks or the task requires it. This avoids slow queries and overwhelming output on large projects.

**Default output directory:** Always use `--output-dir .arize-tmp-traces` on every `ax spans export` call. The CLI automatically creates the directory and adds it to `.gitignore`.

## Prerequisites

Three things are needed: `ax` CLI, an API key (env var or profile), and a space ID. A project name or ID is also needed but usually comes from the user's message.

### Install ax

If `ax` is not installed, not on PATH, or below version `0.3.0`, see ax-setup.md.

### Verify environment

Run a quick check for credentials:

**macOS/Linux (bash):**
```bash
ax --version && echo "--- env ---" && if [ -n "$ARIZE_API_KEY" ]; then echo "ARIZE_API_KEY: (set)"; else echo "ARIZE_API_KEY: (not set)"; fi && echo "ARIZE_SPACE_ID: ${ARIZE_SPACE_ID:-(not set)}" && echo "--- profiles ---" && ax profiles show 2>&1
```

**Windows (PowerShell):**
```powershell
ax --version; Write-Host "--- env ---"; Write-Host "ARIZE_API_KEY: $(if ($env:ARIZE_API_KEY) { '(set)' } else { '(not set)' })"; Write-Host "ARIZE_SPACE_ID: $env:ARIZE_SPACE_ID"; Write-Host "--- profiles ---"; ax profiles show 2>&1
```

**Read the output and proceed immediately** if either the env var or the profile has an API key. Only ask the user if **both** are missing. Resolve failures:

- No API key in env **and** no profile → **AskQuestion**: "Arize API key (https://app.arize.com/admin > API Keys)"
- Space ID unknown → run `ax spaces list -o json` to list all accessible spaces and pick the right one, or **AskQuestion** if the user prefers to provide it directly
- Project unclear → run `ax projects list -l 100 -o json` (add `--space-id` if known), present the names, and ask the user to pick one

**IMPORTANT:** `--space-id` is required when using a human-readable project name as the `PROJECT` positional argument. It is not needed when using a base64-encoded project ID. If you hit `401 Unauthorized` or limit errors when using a project name, resolve it to a base64 ID first (see "Resolving project for export" in Concepts).

## Export Spans: `ax spans export`

The primary command for downloading trace data to a file.

### By trace ID

```bash
ax spans export PROJECT_ID --trace-id TRACE_ID --output-dir .arize-tmp-traces
```

### By span ID

```bash
ax spans export PROJECT_ID --span-id SPAN_ID --output-dir .arize-tmp-traces
```

### By session ID

```bash
ax spans export PROJECT_ID --session-id SESSION_ID --output-dir .arize-tmp-traces
```

### Flags

| Flag | Default | Description |
|------|---------|-------------|
| `PROJECT` (positional) | `$ARIZE_DEFAULT_PROJECT` | Project name or base64 ID |
| `--trace-id` | — | Filter by `context.trace_id` (mutex with other ID flags) |
| `--span-id` | — | Filter by `context.span_id` (mutex with other ID flags) |
| `--session-id` | — | Filter by `attributes.session.id` (mutex with other ID flags) |
| `--filter` | — | SQL-like filter; combinable with any ID flag |
| `--limit, -l` | 500 | Max spans (REST); ignored with `--all` |
| `--space-id` | — | Required when `PROJECT` is a name, or with `--all` |
| `--days` | 30 | Lookback window; ignored if `--start-time`/`--end-time` set |
| `--start-time` / `--end-time` | — | ISO 8601 time range override |
| `--output-dir` | `.arize-tmp-traces` | Output directory |
| `--stdout` | false | Print JSON to stdout instead of file |
| `--all` | false | Unlimited bulk export via Arrow Flight (see below) |

Output is a JSON array of span objects. File naming: `{type}_{id}_{timestamp}/spans.json`.

### Bulk export with `--all`

By default, `ax spans export` is capped at 500 spans by `-l`. Pass `--all` for unlimited bulk export.

```bash
ax spans export PROJECT_ID --space-id SPACE_ID --filter "status_code = 'ERROR'" --all --output-dir .arize-tmp-traces
```

**When to use `--all`:**
- Exporting more than 500 spans
- Downloading full traces with many child spans
- Large time-range exports

**Agent auto-escalation rule:** If an export returns exactly the number of spans requested by `-l` (or 500 if no limit was set), the result is likely truncated. Increase `-l` or re-run with `--all` to get the full dataset — but only when the user asks or the task requires more data.

**Decision tree:**
```
Do you have a --trace-id, --span-id, or --session-id?
├─ YES: count is bounded → omit --all. If result is exactly 500, re-run with --all.
└─ NO (exploratory export):
    ├─ Just browsing a sample? → use -l 50
    └─ Need all matching spans?
        ├─ Expected < 500 → -l is fine
        └─ Expected ≥ 500 or unknown → use --all
            └─ Times out? → batch by --days (e.g., --days 7) and loop
```

**Check span count first:** Before a large exploratory export, check how many spans match your filter:
```bash
# Count matching spans without downloading them
ax spans export PROJECT_ID --filter "status_code = 'ERROR'" -l 1 --stdout | jq 'length'
# If returns 1 (hit limit), run with --all
# If returns 0, no data matches -- check filter or expand --days
```

**Requirements for `--all`:**
- `--space-id` is required (Flight uses `space_id` + `project_name`, not `project_id`)
- `--limit` is ignored when `--all` is set

**Networking notes for `--all`:**
Arrow Flight connects to `flight.arize.com:443` via gRPC+TLS -- this is a different host from the REST API (`api.arize.com`). On internal or private networks, the Flight endpoint may use a different host/port. Configure via:
- ax profile: `flight_host`, `flight_port`, `flight_scheme`
- Environment variables: `ARIZE_FLIGHT_HOST`, `ARIZE_FLIGHT_PORT`, `ARIZE_FLIGHT_SCHEME`

The `--all` flag is also available on `ax traces export`, `ax datasets export`, and `ax experiments export` with the same behavior (REST by default, Flight with `--all`).

## Export Traces: `ax traces export`

Export full traces -- all spans belonging to traces that match a filter. Uses a two-phase approach:

1. **Phase 1:** Find spans matching `--filter` (up to `--limit` via REST, or all via Flight with `--all`)
2. **Phase 2:** Extract unique trace IDs, then fetch every span for those traces

```bash
# Explore recent traces (start small with -l 50, pull more if needed)
ax traces export PROJECT_ID -l 50 --output-dir .arize-tmp-traces

# Export traces with error spans (REST, up to 500 spans in phase 1)
ax traces export PROJECT_ID --filter "status_code = 'ERROR'" --stdout

# Export all traces matching a filter via Flight (no limit)
ax traces export PROJECT_ID --space-id SPACE_ID --filter "status_code = 'ERROR'" --all --output-dir .arize-tmp-traces
```

### Flags

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `PROJECT` | string | required | Project name or base64 ID (positional arg) |
| `--filter` | string | none | Filter expression for phase-1 span lookup |
| `--space-id` | string | none | Space ID; required when `PROJECT` is a name or when using `--all` (Arrow Flight) |
| `--limit, -l` | int | 50 | Max number of traces to export |
| `--days` | int | 30 | Lookback window in days |
| `--start-time` | string | none | Override start (ISO 8601) |
| `--end-time` | string | none | Override end (ISO 8601) |
| `--output-dir` | string | `.` | Output directory |
| `--stdout` | bool | false | Print JSON to stdout instead of file |
| `--all` | bool | false | Use Arrow Flight for both phases (see spans `--all` docs above) |
| `-p, --profile` | string | default | Configuration profile |

### How it differs from `ax spans export`

- `ax spans export` exports individual spans matching a filter
- `ax traces export` exports complete traces -- it finds spans matching the filter, then pulls ALL spans for those traces (including siblings and children that may not match the filter)

## Filter Syntax Reference

SQL-like expressions passed to `--filter`.

### Common filterable columns

| Column | Type | Description | Example Values |
|--------|------|-------------|----------------|
| `name` | string | Span name | `'ChatCompletion'`, `'retrieve_docs'` |
| `status_code` | string | Status | `'OK'`, `'ERROR'`, `'UNSET'` |
| `latency_ms` | number | Duration in ms | `100`, `5000` |
| `parent_id` | string | Parent span ID | null for root spans |
| `context.trace_id` | string | Trace ID | |
| `context.span_id` | string | Span ID | |
| `attributes.session.id` | string | Session ID | |
| `attributes.openinference.span.kind` | string | Span kind | `'LLM'`, `'CHAIN'`, `'TOOL'`, `'AGENT'`, `'RETRIEVER'`, `'RERANKER'`, `'EMBEDDING'`, `'GUARDRAIL'`, `'EVALUATOR'` |
| `attributes.llm.model_name` | string | LLM model | `'gpt-4o'`, `'claude-3'` |
| `attributes.input.value` | string | Span input | |
| `attributes.output.value` | string | Span output | |
| `attributes.error.type` | string | Error type | `'ValueError'`, `'TimeoutError'` |
| `attributes.error.message` | string | Error message | |
| `event.attributes` | string | Error tracebacks | Use CONTAINS (not exact match) |

### Operators

`=`, `!=`, `<`, `<=`, `>`, `>=`, `AND`, `OR`, `IN`, `CONTAINS`, `LIKE`, `IS NULL`, `IS NOT NULL`

### Examples

```
status_code = 'ERROR'
latency_ms > 5000
name = 'ChatCompletion' AND status_code = 'ERROR'
attributes.llm.model_name = 'gpt-4o'
attributes.openinference.span.kind IN ('LLM', 'AGENT')
attributes.error.type LIKE '%Transport%'
event.attributes CONTAINS 'TimeoutError'
```

### Tips

- Prefer `IN` over multiple `OR` conditions: `name IN ('a', 'b', 'c')` not `name = 'a' OR name = 'b' OR name = 'c'`
- Start broad with `LIKE`, then switch to `=` or `IN` once you know exact values
- Use `CONTAINS` for `event.attributes` (error tracebacks) -- exact match is unreliable on complex text
- Always wrap string values in single quotes

## Workflows

### Debug a failing trace

1. `ax traces export PROJECT_ID --filter "status_code = 'ERROR'" -l 50 --output-dir .arize-tmp-traces`
2. Read the output file, look for spans with `status_code: ERROR`
3. Check `attributes.error.type` and `attributes.error.message` on error spans

### Download a conversation session

1. `ax spans export PROJECT_ID --session-id SESSION_ID --output-dir .arize-tmp-traces`
2. Spans are ordered by `start_time`, grouped by `context.trace_id`
3. If you only have a trace_id, export that trace first, then look for `attributes.session.id` in the output to get the session ID

### Export for offline analysis

```bash
ax spans export PROJECT_ID --trace-id TRACE_ID --stdout | jq '.[]'
```


## Span Column Reference (OpenInference Semantic Conventions)

### Core Identity and Timing

| Column | Description |
|--------|-------------|
| `name` | Span operation name (e.g., `ChatCompletion`, `retrieve_docs`) |
| `context.trace_id` | Trace ID -- all spans in a trace share this |
| `context.span_id` | Unique span ID |
| `parent_id` | Parent span ID. `null` for root spans (= traces) |
| `start_time` | When the span started (ISO 8601) |
| `end_time` | When the span ended |
| `latency_ms` | Duration in milliseconds |
| `status_code` | `OK`, `ERROR`, `UNSET` |
| `status_message` | Optional message (usually set on errors) |
| `attributes.openinference.span.kind` | `LLM`, `CHAIN`, `TOOL`, `AGENT`, `RETRIEVER`, `RERANKER`, `EMBEDDING`, `GUARDRAIL`, `EVALUATOR` |

### Where to Find Prompts and LLM I/O

**Generic input/output (all span kinds):**

| Column | What it contains |
|--------|-----------------|
| `attributes.input.value` | The input to the operation. For LLM spans, often the full prompt or serialized messages JSON. For chain/agent spans, the user's question. |
| `attributes.input.mime_type` | Format hint: `text/plain` or `application/json` |
| `attributes.output.value` | The output. For LLM spans, the model's response. For chain/agent spans, the final answer. |
| `attributes.output.mime_type` | Format hint for output |

**LLM-specific message arrays (structured chat format):**

| Column | What it contains |
|--------|-----------------|
| `attributes.llm.input_messages` | Structured input messages array (system, user, assistant, tool). **Where chat prompts live** in role-based format. |
| `attributes.llm.input_messages.roles` | Array of roles: `system`, `user`, `assistant`, `tool` |
| `attributes.llm.input_messages.contents` | Array of message content strings |
| `attributes.llm.output_messages` | Structured output messages from the model |
| `attributes.llm.output_messages.contents` | Model response content |
| `attributes.llm.output_messages.tool_calls.function.names` | Tool calls the model wants to make |
| `attributes.llm.output_messages.tool_calls.function.arguments` | Arguments for those tool calls |

**Prompt templates:**

| Column | What it contains |
|--------|-----------------|
| `attributes.llm.prompt_template.template` | The prompt template with variable placeholders (e.g., `"Answer {question} using {context}"`) |
| `attributes.llm.prompt_template.variables` | Template variable values (JSON object) |

**Finding prompts by span kind:**

- **LLM span**: Check `attributes.llm.input_messages` for structured chat messages, OR `attributes.input.value` for serialized prompt. Check `attributes.llm.prompt_template.template` for the template.
- **Chain/Agent span**: Check `attributes.input.value` for the user's question. Actual LLM prompts are on child LLM spans.
- **Tool span**: Check `attributes.input.value` for tool input, `attributes.output.value` for tool result.

### LLM Model and Cost

| Column | Description |
|--------|-------------|
| `attributes.llm.model_name` | Model identifier (e.g., `gpt-4o`, `claude-3-opus-20240229`) |
| `attributes.llm.invocation_parameters` | Model parameters JSON (temperature, max_tokens, top_p, etc.) |
| `attributes.llm.token_count.prompt` | Input token count |
| `attributes.llm.token_count.completion` | Output token count |
| `attributes.llm.token_count.total` | Total tokens |
| `attributes.llm.cost.prompt` | Input cost in USD |
| `attributes.llm.cost.completion` | Output cost in USD |
| `attributes.llm.cost.total` | Total cost in USD |

### Tool Spans

| Column | Description |
|--------|-------------|
| `attributes.tool.name` | Tool/function name |
| `attributes.tool.description` | Tool description |
| `attributes.tool.parameters` | Tool parameter schema (JSON) |

### Retriever Spans

| Column | Description |
|--------|-------------|
| `attributes.retrieval.documents` | Retrieved documents array |
| `attributes.retrieval.documents.ids` | Document IDs |
| `attributes.retrieval.documents.scores` | Relevance scores |
| `attributes.retrieval.documents.contents` | Document text content |
| `attributes.retrieval.documents.metadatas` | Document metadata |

### Reranker Spans

| Column | Description |
|--------|-------------|
| `attributes.reranker.query` | The query being reranked |
| `attributes.reranker.model_name` | Reranker model |
| `attributes.reranker.top_k` | Number of results |
| `attributes.reranker.input_documents.*` | Input documents (ids, scores, contents, metadatas) |
| `attributes.reranker.output_documents.*` | Reranked output documents |

### Session, User, and Custom Metadata

| Column | Description |
|--------|-------------|
| `attributes.session.id` | Session/conversation ID -- groups traces into multi-turn sessions |
| `attributes.user.id` | End-user identifier |
| `attributes.metadata.*` | Custom key-value metadata. Any key under this prefix is user-defined (e.g., `attributes.metadata.user_email`). Filterable. |

### Errors and Exceptions

| Column | Description |
|--------|-------------|
| `attributes.exception.type` | Exception class name (e.g., `ValueError`, `TimeoutError`) |
| `attributes.exception.message` | Exception message text |
| `event.attributes` | Error tracebacks and detailed event data. Use `CONTAINS` for filtering. |

### Evaluations and Annotations

| Column | Description |
|--------|-------------|
| `annotation.<name>.label` | Human or auto-eval label (e.g., `correct`, `incorrect`) |
| `annotation.<name>.score` | Numeric score (e.g., `0.95`) |
| `annotation.<name>.text` | Freeform annotation text |

### Embeddings

| Column | Description |
|--------|-------------|
| `attributes.embedding.model_name` | Embedding model name |
| `attributes.embedding.texts` | Text chunks that were embedded |

## Troubleshooting

| Problem | Solution |
|---------|----------|
| `ax: command not found` | See ax-setup.md |
| `SSL: CERTIFICATE_VERIFY_FAILED` | macOS: `export SSL_CERT_FILE=/etc/ssl/cert.pem`. Linux: `export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt`. Windows: `$env:SSL_CERT_FILE = (python -c "import certifi; print(certifi.where())")` |
| `No such command` on a subcommand that should exist | The installed `ax` is outdated. Reinstall: `uv tool install --force --reinstall arize-ax-cli` (requires shell access to install packages) |
| `No profile found` | Follow "Resolve credentials" in Prerequisites to auto-discover or prompt for the API key |
| `401 Unauthorized` with valid API key | You are likely using a project name without `--space-id`. Add `--space-id SPACE_ID`, or resolve to a base64 project ID first: `ax projects list --space-id SPACE_ID -l 100 -o json` and use the project's `id`. |
| `No spans found` | Expand `--days` (default 30), verify project ID |
| `Filter error` or `invalid filter expression` | Check column name spelling (e.g., `attributes.openinference.span.kind` not `span_kind`), wrap string values in single quotes, use `CONTAINS` for free-text fields |
| `unknown attribute` in filter | The attribute path is wrong or not indexed. Try browsing a small sample first to see actual column names: `ax spans export PROJECT_ID -l 5 --stdout \| jq '.[0] \| keys'` |
| `Timeout on large export` | Use `--days 7` to narrow the time range |

## Related Skills

- **arize-dataset**: After collecting trace data, create labeled datasets for evaluation → use `arize-dataset`
- **arize-experiment**: Run experiments comparing prompt versions against a dataset → use `arize-experiment`
- **arize-prompt-optimization**: Use trace data to improve prompts → use `arize-prompt-optimization`
- **arize-link**: Turn trace IDs from exported data into clickable Arize UI URLs → use `arize-link`

## Save Credentials

At session end, if the user manually provided an API key, space ID, or project ID (not loaded from an existing profile), offer to save them to `~/.arize/config.toml`. Use **AskQuestion** with "Yes, save them" / "No thanks". Skip if all values were already in the profile.

Read the existing file (or create it), add/update only the new fields, and write it back:

```toml
[auth]
api_key = "THE_API_KEY"

[defaults]
space_id = "THE_SPACE_ID"
project = "THE_PROJECT_NAME"
```