Ax Agent Observability

---
name: ax-agent-observability
description: This skill helps an LLM generate correct AxAgent observability code using @ax-llm/ax. Use when the user asks about axGlobals.onUsage, usageContext, centralized or multi-tenant usage accounting, actorTurnCallback, onContextEvent, agentStatusCallback, onFunctionCall, reportSuccess, reportFailure, getChatLog(), getUsage(), resetUsage(), debug traces, progress updates, or telemetry for AxAgent runs.
version: "25.1.0"
---

# AxAgent Observability Rules (@ax-llm/ax)

Use this skill when an agent needs runtime visibility, progress reporting, tracing, usage accounting, or chat-log access. For ordinary agent setup use `ax-agent`. For RLM runtime policy use `ax-agent-rlm`. For memories and dynamic skill loading use `ax-agent-memory-skills`.

## Choose The Smallest Hook

- Need a quick prompt/runtime trace during development -> start with `debug: true`.
- Need structured per-turn code, raw runtime result, formatted output, provider thoughts, or actor stage -> use `actorTurnCallback`.
- Need context-pressure and compaction telemetry -> use `onContextEvent`.
- Need real-time task progress emitted by actor code -> use `agentStatusCallback`.
- Need every runtime function call before execution -> use `onFunctionCall`.
- Need model prompts/responses after a run -> use `getChatLog()`.
- Need centralized chat/embed usage across APIs, users, agents, and services -> use `axGlobals.onUsage` plus `usageContext`.
- Need token usage by actor/responder -> use `getUsage()` and `resetUsage()`.
- Need usage split by context and task stages -> use `getStagedUsage()`.
- Need Ax program traces -> use `getTraces()`.
- Do not add multiple hooks unless the user clearly needs each output stream.

## Global Runtime Defaults

OpenTelemetry and debug defaults come from the shared Ax runtime surface:

```typescript
import { axGlobals, axCreateDefaultColorLogger } from '@ax-llm/ax';
import { metrics, trace } from '@opentelemetry/api';

axGlobals.rateLimiter = async (next, info) => next();
axGlobals.tracer = trace.getTracer('agent-app');
axGlobals.meter = metrics.getMeter('agent-app');
axGlobals.debug = true;
axGlobals.logger = axCreateDefaultColorLogger();
axGlobals.onUsage = (event) => usageQueue.enqueue(event);
```

Each agent run snapshots these globals. A forward-scoped `rateLimiter`, `tracer`, or `meter` overrides agent defaults, child-generator defaults, service hooks, and globals for that invocation. Ax carries it through the distiller, executor, responder, repeated actor turns, citation repair, built-in `llmQuery`, context-map work, checkpoint/tombstone summaries, and other direct internal model calls without mutating child programs or leaking across concurrent runs.

The limiter wraps model execution and its failures propagate. Tracer, meter, and usage-observer failures are fail-open. Runtime-hook spans contain metadata only—never prompts, outputs, tool arguments, or tool results—and preserve agent → internal AxGen → provider/tool parentage. External meter instruments do not replace or derive from balancer-local `getMetrics()` snapshots. Use AxAgent callbacks below when the caller needs structured agent-turn events rather than spans or metrics.

## Centralized Usage Observer

Use the process-wide usage observer for application accounting across many agents, API routes, tenants, and users. Keep `getUsage()` for inspecting one agent instance after a run.

```typescript
import { axGlobals } from '@ax-llm/ax';

axGlobals.onUsage = (event) => {
  usageQueue.enqueue(event); // Must return immediately.
};

await supportAgent.forward(
  llm,
  { query: request.body.query },
  {
    usageContext: {
      tenantId: auth.tenantId,
      userId: auth.userId,
      requestId: request.id,
      runId: crypto.randomUUID(),
      feature: 'support-chat',
      attributes: { environment: 'production' },
    },
  }
);
```

Rules:

- The observer receives one immutable normalized event for each completed chat or embedding call that reports provider usage. A fully consumed stream emits once; an unconsumed or cancelled stream may not emit.
- Events include `operation`, `ai`, `model`, normalized `tokens`, `streaming`, optional `context`, and available session or remote request IDs.
- Put stable attribution such as application or environment in AI service-level `usageContext`. Put tenant, user, request, run, and feature attribution in per-call or per-forward `usageContext`.
- Per-call context overrides service defaults. `attributes` are shallow-merged.
- The observer is best-effort and fail-open. Ax does not await it and ignores observer failures, so synchronously enqueue and persist or aggregate out of band.
- The registration is process-wide. A new assignment replaces the previous observer; set `axGlobals.onUsage = undefined` during test teardown or shutdown when appropriate.
- In multi-process or serverless deployments, send events to a shared durable pipeline. Do not treat an in-memory total as application-wide accounting.
- Keep identifiers opaque and attributes low-cardinality. Avoid prompts, responses, secrets, and other sensitive payloads.
- Token events do not estimate currency cost. Apply a versioned provider/model pricing table downstream.

For direct AI calls and the complete event shape, also read `ax-ai`.

## Actor Turn Callback

Use `actorTurnCallback` when the caller needs structured telemetry for each actor turn.

What it gives you:

- `code`: the normalized JavaScript code the actor produced
- `stage`: which actor produced the turn (`distiller` or `executor`)
- `result`: the raw untruncated runtime return value from executing that code
- `output`: the formatted action-log output string after Ax normalizes and truncates it for prompt replay
- `thought`: the actor model's `thought` field when `showThoughts` is enabled and the provider returns one
- `executorResult`: the full actor payload returned by the current actor stage, kept under this historical field name for compatibility
- `isError`: whether the execution path for that turn was treated as an error
- `usage`: token usage for this actor turn only
- `model`: model used for this turn when explicitly set through `executorModelPolicy`
- `chatLogMessages`: raw ChatML conversation for this turn, populated only when an actor turn callback is set

Use it for:

- debug UIs that want to show code plus raw runtime results
- tracing and analytics
- capturing `thought` for internal diagnostics when supported by the provider
- storing per-turn execution artifacts without scraping the prompt/action log

Important:

- `output` is not raw stdout; it is the formatted replay string used in the action log.
- `result` is the raw runtime result before Ax applies type-aware serialization and budget-proportional truncation.
- `thought` is optional and only appears when the underlying `AxGen` call had `showThoughts` enabled and the provider actually returned a thought field.
- `actionLogEntryCount` and `guidanceLogEntryCount` reflect the live log sizes after the turn is processed, including resumed runs.
- `actorTurnCallback` fires for the configured agent instance. Child agents passed through `functions: [...]` should define their own callback if you need their internal actor turns; use `onFunctionCall` on the parent to observe the parent-side child-agent invocation.

Good pattern:

```typescript
const supportAgent = agent('query:string -> answer:string', {
  contextFields: ['query'],
  runtime,
  actorTurnCallback: ({
    stage,
    turn,
    actionLogEntryCount,
    guidanceLogEntryCount,
    code,
    result,
    output,
    thought,
    isError,
    usage,
    model,
  }) => {
    console.log({
      turn,
      stage,
      model,
      actionLogEntryCount,
      guidanceLogEntryCount,
      isError,
      code,
      rawResult: result,
      replayOutput: output,
      thought,
      usage,
    });
  },
  executorOptions: {
    model: 'gpt-5.4-mini',
    showThoughts: true,
  },
});
```

Callback type:

```typescript
actorTurnCallback?: (turn: {
  stage: 'distiller' | 'executor';
  turn: number;
  actionLogEntryCount: number;
  guidanceLogEntryCount: number;
  executorResult: Record<string, unknown>;
  code: string;
  result: unknown;
  output: string;
  isError: boolean;
  thought?: string;
  usage?: AxProgramUsage[];
  model?: string;
  chatLogMessages?: ReadonlyArray<{ role: string; content: string }>;
}) => void | Promise<void>;

actorTurnCallback?: (turn: {
  stage: 'distiller' | 'executor';
  turn: number;
  actionLogEntryCount: number;
  guidanceLogEntryCount: number;
  executorResult: Record<string, unknown>;
  code: string;
  result: unknown;
  output: string;
  isError: boolean;
  thought?: string;
  usage?: AxProgramUsage[];
  model?: string;
  chatLogMessages?: ReadonlyArray<{ role: string; content: string }>;
}) => void | Promise<void>; // deprecated alias
```

## Context Event Observability

Use `onContextEvent` when the caller needs structured telemetry about prompt pressure and compaction. It does not change model behavior directly; it is for logs, evals, and dashboards.

Events:

- `budget_check`: character-based prompt pressure before an actor turn, with detailed metrics kept out of the actor prompt
- `checkpoint_created` / `checkpoint_cleared`: checkpoint lifecycle events with covered turns and reason
- `tombstone_created`: compact resolved-error summary creation
- `relevance_ranking`: emitted once per ranked domain per forward when `relevanceRanking` is enabled; carries `domain` (`'modules' | 'skills' | 'memories'`), the `shortlist` (`{ id, score }[]`, most relevant first), and `suppressed` (true when the low-confidence guard emitted no hint)
- `field_auto_promoted`: emitted once per field per run when `autoUpgrade` keeps an oversized undeclared input value runtime-only; carries `fieldName`, `originalChars`, and `promptPreviewChars` (undefined when no inline preview was kept)

To measure whether the advisory hint helps, join per forward: `relevance_ranking.shortlist` ids against what the actor then loaded — for modules the internal `discover` calls (`onFunctionCall` with `kind: 'internal'`, `name: 'discover'`, `args.request`) plus the module part of external `qualifiedName`s; for skills `onLoadedSkills` / `used(id)`; for memories `onLoadedMemories` / `used(id)`.

Rules:

- `contextPressure` in the actor prompt is intentionally compact (`ok`, `watch`, `critical` plus one short instruction).
- Budget metrics are character-based for provider neutrality and are exposed through `onContextEvent`, not the actor prompt.
- Callback errors are swallowed so telemetry cannot break the agent run.
- Do not scrape actor prompts for pressure metrics.

```typescript
const supportAgent = agent('query:string -> answer:string', {
  contextFields: ['query'],
  runtime,
  contextPolicy: { preset: 'checkpointed', budget: 'balanced' },
  onContextEvent: (event) => {
    if (event.kind === 'budget_check') {
      console.log(event.pressure, event.mutablePromptChars);
    }
  },
});
```

Type:

```typescript
onContextEvent?: (event: AxAgentContextEvent) => void | Promise<void>;
```

## Agent Status Callback

Use `agentStatusCallback` when the caller wants real-time progress updates from the actor. When set, the actor can call `await reportSuccess(message)` and `await reportFailure(message)` in its JavaScript turns.

```typescript
const supportAgent = agent('query:string -> answer:string', {
  contextFields: ['query'],
  runtime,
  agentStatusCallback: (message, status) => {
    console.log(`[${status}] ${message}`);
  },
});
```

Rules:

- `agentStatusCallback` receives `(message: string, status: 'success' | 'failed')`.
- When set, the actor prompt automatically includes `reportSuccess(message)` and `reportFailure(message)` as available runtime functions.
- The actor is instructed to keep the user updated on task progress.
- `reportSuccess` and `reportFailure` are reserved runtime names when the callback is configured.
- Child agents inherit the callback via the RLM config.

Type:

```typescript
agentStatusCallback?: (
  message: string,
  status: 'success' | 'failed'
) => void | Promise<void>;
```

## On Function Call

Use `onFunctionCall` when the caller wants to observe every function call the actor makes from the JS runtime. It fires before the underlying function runs.

```typescript
const supportAgent = agent('query:string -> answer:string', {
  contextFields: ['query'],
  runtime,
  functions: [helperAgent, lookupOrderTool],
  onFunctionCall: ({ name, qualifiedName, args, kind }) => {
    console.log(`[${kind}] ${qualifiedName}`, args);
  },
});
```

Rules:

- Receives `{ name, qualifiedName, args, kind }`.
- `name` is the bare function name, e.g. `'lookupOrder'`.
- `qualifiedName` is the namespaced name as the actor sees it, e.g. `'tools.lookupOrder'`; for un-namespaced runtime globals it equals `name`.
- `args` is the resolved positional/named arguments object (`Record<string, unknown>`).
- `kind` is `'external'` for caller-registered `functions`.
- `kind` is `'internal'` for agent-injected globals: child agents, `discover`, `recall`, and `used`.
- Fires once per call, before the function executes.
- Errors thrown inside the callback are swallowed so they cannot break the actor loop.
- This is independent from the DSP-layer `onFunctionCall` on `AxProgramForwardOptions`; that hook is for LLM tool-calls and never fires under AxAgent because AxAgent injects functions as runtime globals.

Type:

```typescript
onFunctionCall?: (call: {
  name: string;
  qualifiedName: string;
  args: Record<string, unknown>;
  kind: 'internal' | 'external';
}) => void | Promise<void>;
```

## Chat Log, Usage, And Traces

`AxAgent` exposes actor and responder sub-programs. `getChatLog()` returns the same flat `AxChatLogEntry[]` shape as `AxGen` and `AxFlow`; use each entry's optional `name` field to distinguish `distiller`, `executor`, and `responder`. `getUsage()` returns token usage split by actor/responder.

`getChatLog()`, `getUsage()`, and trace export refresh the agent-level snapshot
from the distiller, executor, and responder before returning. This remains true
after `forward()` throws, so successful model calls made before an actor limit,
provider error, cancellation, or responder error are still observable and are
not silently dropped from usage accounting.

### getChatLog()

Returns the full normalized chat history after any `.forward()` call. Each entry is one `ai.chat()` round-trip. Actor stages accumulate one entry per turn; the responder typically has one entry.

```typescript
const log = myAgent.getChatLog();

for (const entry of log) {
  console.log(entry.name, entry.model);
  for (const msg of entry.messages) {
    console.log(`[${msg.role}]`, msg.content);
  }
}
```

Each `AxChatLogEntry` captures the full prompt sent to the model and its response:

```typescript
type AxChatLogMessage =
  | { role: 'system'; content: string }
  | { role: 'user'; content: string }
  | { role: 'assistant'; content: string }
  | { role: 'tool'; name: string; content: string };

type AxChatLogEntry = {
  name?: string; // e.g. "distiller", "executor", "responder"
  model: string;
  messages: AxChatLogMessage[];
  modelUsage?: AxProgramUsage;
  stage?: 'ctx' | 'task';
};
```

### getUsage()

Returns token usage split by actor/responder. Each sub-array contains one `AxProgramUsage` entry per model/run, merged by `(ai, model)` key.

```typescript
const usage = myAgent.getUsage();
// { actor: AxProgramUsage[], responder: AxProgramUsage[] }

console.log('Actor tokens:', usage.actor[0]?.tokens);
console.log('Responder tokens:', usage.responder[0]?.tokens);
```

### getStagedUsage()

Returns usage split by pipeline stage. The `ctx` stage has the distiller actor only; the `task` stage has the executor actor plus responder.

```typescript
const staged = myAgent.getStagedUsage();
console.log(staged.ctx?.actor);
console.log(staged.task.actor);
console.log(staged.task.responder);
```

### getTraces()

Returns Ax program traces for the agent pipeline. Use it when the caller needs trace data rather than chat messages or token summaries.

```typescript
const traces = myAgent.getTraces();
```

### resetUsage()

Resets both actor and responder usage at once:

```typescript
myAgent.resetUsage();
```

Type signatures:

```typescript
// AxAgent
agent.getChatLog(): readonly AxChatLogEntry[]
agent.getUsage(): { actor: AxProgramUsage[]; responder: AxProgramUsage[] }
agent.getStagedUsage(): { ctx?: AxAgentUsage; task: AxAgentUsage }
agent.getTraces(): AxProgramTrace[]
agent.resetUsage(): void

// AxGen / AxFlow
gen.getChatLog(): readonly AxChatLogEntry[]
gen.getUsage(): AxProgramUsage[]
```

## Do Not Generate

- Do not add both `debug: true` and `actorTurnCallback` unless the user wants both unstructured prompt/runtime visibility and structured telemetry.
- Do not scrape actor prompts or action logs when a callback exposes the data directly.
- Do not let observability callback failures break the agent run; Ax swallows callback errors for telemetry hooks.
- Do not use DSP-layer `onFunctionCall` when the user wants AxAgent runtime function calls.
- Do not enable `showThoughts` unless the user needs provider thought diagnostics and the provider supports it.
- Do not use `getUsage()` as the centralized source of truth across shared agents or processes.
- Do not perform database or network work inline in `axGlobals.onUsage`; enqueue and return.