Neuron Frontend Integration

---
name: neuron-frontend-integration
description: Connect a Neuron AI agent to a real frontend that executes tools in the browser — declaring deferred (frontend) tools, building the endpoint that serves a new turn or continues a suspended run, translating tool results and approval decisions coming from the Vercel AI SDK (useChat), the official AG-UI client, or CopilotKit (useFrontendTool, useInterrupt, runtime bridge), and handling partial results, retries, reloads, restarts and errors. Use this skill whenever the user mentions frontend tools, client-side tools, browser tools, DeferredTool, tool results from the frontend, AG-UI, CopilotKit, Vercel AI SDK, useChat, addToolOutput, addToolApprovalResponse, sendAutomaticallyWhen, useFrontendTool, useInterrupt, HttpAgent, AGUIInputTranslator, VercelAIInputTranslator, submitToolResults, submitApprovalDecisions, submitInputs with a frontend payload, or asks which versions of these libraries Neuron supports.
---

# Neuron AI Frontend Integration

This skill covers the round trip between a Neuron agent and a frontend that executes some of the agent's tools: the backend endpoint, the inbound translation of what the client sends, and the exact client-side wiring for the Vercel AI SDK, the official AG-UI client, and CopilotKit.

Every example and every behavioural claim below is proven by the integration suite in `tests/Integration/Frontend/` (run it with `composer test:frontend`). If you change how the clients are wired, prove the change there first.

## The Mental Model

A **deferred tool** is a tool the model may call but the backend cannot execute. When the model calls one, `ToolNode` executes the local tools of the batch, then `AwaitToolResultsNode` **suspends the run** with a `ToolResultsRequest` carrying the calls that wait for the frontend. The stream adapter (`AGUIAdapter` or `VercelAIAdapter`, see the **neuron-streaming** skill for how adapters shape output and how channels push it) publishes those calls in the protocol's vocabulary, the browser executes them, and the client library sends the outcomes back over HTTP. The endpoint rebuilds the agent and continues with `submitToolResults($results)->events()` for a native result map. Raw SDK envelopes use `submitInputs($payload, $translator)->events()` so the protocol translator can extract the responses. The next inference sees the results as ordinary tool result messages.

Three identities, never interchangeable:

| Identity | What it is | Who owns it |
|---|---|---|
| Thread | The conversation and the workflow ID. AG-UI `threadId`, Vercel chat `id` | Application |
| Call ID | The only identity of a tool result. Results, approvals and retries are matched by it | Model / provider |
| AG-UI `runId` | A per-request identifier of the client. **Not** Neuron's durable run ID | Client |

Two rules decide correctness:

- **The request shape decides new turn versus continuation.** A user message starts a turn; tool results, approval responses or an AG-UI `resume` array continue one. An invalid continuation is an error response, never a fallback to a new turn.
- **Results accumulate by call ID.** A partial batch re-suspends with the accepted results kept, an identical repeat is accepted, a different outcome for a settled call is refused, and a submission that belongs to a completed run or an earlier turn is refused.

## Backend

### Declaring frontend tools

A `FrontendTool` is a declaration only: name, description and an optional JSON schema. Executing it on the backend throws.

```php
use NeuronAI\Tools\FrontendTool;

new FrontendTool(
    name: 'read_title',
    description: 'Read the title of the page the user is looking at.'
);

new FrontendTool(
    name: 'read_text',
    description: 'Read the text of an element on the page.',
    inputSchema: [
        'type' => 'object',
        'properties' => ['selector' => ['type' => 'string', 'description' => 'CSS selector']],
        'required' => ['selector'],
    ]
);
```

Where the catalog comes from differs by protocol:

- **Vercel** has no client-supplied catalog. The backend declares the deferred tools and the frontend implements handlers with the same names.
- **AG-UI and CopilotKit** send the catalog in `RunAgentInput.tools` on every request. Build it with `AGUIInputTranslator::tools($payload)` and apply your tool policy before attaching: reject names that shadow backend tools, and gate approval where you want it. Schemas generated by CopilotKit from zod (nested objects, arrays, optional and nullable properties, enums) and `type: ["string", "null"]` are accepted by the converter.

A deferred tool can require approval like any other: `$tool->requireApproval()`. The run then suspends first with an `ApprovalRequest`; after approval it suspends again with the `ToolResultsRequest`, and only then does the frontend handler run. A rejection never reaches the frontend: the call is stamped with an instruction string for the model and inference continues.

### Native application payloads

For a custom frontend that sends maps keyed by tool call ID, use the Agent methods
directly. No event name, interruption ID or native translator is required:

```php
// Answer the current tool approval request.
$state = $agent->submitApprovalDecisions([
    'call_123' => 'approve',
    'call_456' => ['reject', 'Do not share this page'],
])->run();

// On a later request, after the approved frontend tool has executed:
$frames = $agent->submitToolResults([
    'call_123' => ['result' => ['title' => 'Example']],
])->events();
```

Each result entry contains exactly one `result` (any JSON-compatible value,
including `false`, `0` and `null`) or `error` (a string). For example,
`['call_123' => ['error' => 'Browser operation cancelled']]` reports execution
failure. Partial results are retained and the agent waits for the remaining calls.
Approval and execution are separate phases: a result cannot answer an approval.

Both methods stage a continuation. Finish with `run()` for `AgentState` or
`events()` to iterate its output; either way, the answer streams only if the turn
started with `stream()` or `chat(..., stream: true)`. Do not call `chat()` or
`stream()` to deliver these maps.
Reconstruct the same thread, durable history and persistence on every request.
Missing runs, unknown call IDs and invalid maps fail before execution.

### Raw SDK payloads

AG-UI `messages`/`resume` and Vercel `messages`/`parts` are protocol envelopes,
not native result maps. Keep the corresponding protocol translator in the
endpoints below. Do not pass the raw envelope to `submitToolResults()` or
`submitApprovalDecisions()`. If an application already produces a native map,
use the methods above instead of constructing a native translator.

### The endpoint

Reconstruct a fresh agent on every request from durable persistence and a durable message store; nothing may depend on the previous PHP object.

```php
use NeuronAI\Agent\Agent;
use NeuronAI\Chat\History\SQLMessageStore;
use NeuronAI\Workflow\Persistence\DatabasePersistence;

$agent = Agent::make(workflowId: $threadId)
    ->setMessageStore(new SQLMessageStore($pdo))
    ->setPersistence(new DatabasePersistence($pdo))
    ->setAiProvider($provider)
    ->addTool($backendTools);

foreach ($frontendTools as $tool) {
    if (in_array($tool->getName(), $backendToolNames, true)) {
        throw new RuntimeException("Frontend tool '{$tool->getName()}' collides with a backend tool.");
    }
    $agent->addTool($tool);
}
```

**AG-UI endpoint.** The thread is `threadId`; the adapter is seeded with the client's `runId`, messages and state so snapshots are complete. A `resume` array or a trailing tool message is a continuation; a trailing user message is a new turn.

```php
use NeuronAI\Agent\Adapters\AGUIAdapter;
use NeuronAI\Agent\Frontend\AGUIInputTranslator;
use NeuronAI\Chat\Messages\UserMessage;

$messages = $payload['messages'] ?? [];
$last = $messages === [] ? null : $messages[array_key_last($messages)];

$translator = new AGUIInputTranslator();
$agent = buildAgent($payload['threadId'], $translator->tools($payload));

$adapter = new AGUIAdapter($payload['threadId'], $payload['runId'] ?? null, $messages, $payload['state'] ?? []);
$agent->setStreamAdapter(fn (): AGUIAdapter => $adapter);

$continuation = ($payload['resume'] ?? []) !== [] || ($last['role'] ?? null) === 'tool';

if ($continuation) {
    $frames = $agent->submitInputs($payload, $translator)->events();
} elseif (($last['role'] ?? null) === 'user') {
    $frames = $agent->stream(new UserMessage((string) $last['content']));
} else {
    throw new InputTranslationException('AG-UI input must end with a user message or carry a continuation.');
}
```

**Vercel endpoint.** The thread is the chat `id` that `useChat({ id })` sends. A trailing assistant message is a continuation; reuse its message ID and parts on the adapter so already dispatched parts are not dispatched twice and frontend-owned results are not echoed back.

```php
use NeuronAI\Agent\Adapters\VercelAIAdapter;
use NeuronAI\Agent\Frontend\VercelAIInputTranslator;

$agent = buildAgent($payload['id'], $frontendTools);

if (($last['role'] ?? null) === 'assistant') {
    $adapter = new VercelAIAdapter($last['id'], $last['parts'] ?? []);
    $agent->setStreamAdapter(fn (): VercelAIAdapter => $adapter);
    $frames = $agent->submitInputs($payload, new VercelAIInputTranslator())->events();
} elseif (($last['role'] ?? null) === 'user') {
    $adapter = new VercelAIAdapter();
    $agent->setStreamAdapter(fn (): VercelAIAdapter => $adapter);
    $text = implode('', array_map(
        fn (array $part): string => $part['type'] === 'text' ? (string) $part['text'] : '',
        $last['parts'] ?? [],
    ));
    $frames = $agent->stream(new UserMessage($text));
}
```

**Streaming and errors.** This is the outbound half described in the **neuron-streaming** skill: send the adapter's headers (`getHeaders()` includes `x-vercel-ai-ui-message-stream: v1` for Vercel), then frame each `ProtocolEvent` with `SSEEncoder` and flush after each line. Progress events yielded by nodes and pushed delivery through a channel work unchanged behind these endpoints. `submitInputs()` validates eagerly, before the first frame, so translation failures become normal HTTP errors. A failure after the first frame is already on the wire as the adapter's own error frame (`RUN_ERROR` or an `error` part); catch it and stop.

```php
use NeuronAI\Workflow\Streaming\SSEEncoder;

foreach ($adapter->getHeaders() as $name => $value) {
    header("{$name}: {$value}");
}
try {
    foreach (SSEEncoder::encode($frames) as $frame) {
        echo $frame;
        flush();
    }
} catch (Throwable) {
}
```

Map the eager failures to statuses the client can act on:

| Exception | Meaning | Status |
|---|---|---|
| `InputTranslationException` | No persisted run, no matching input, malformed payload | 400 |
| `WorkflowException` | Conflicting result for a settled call, stale run identity | 409 |
| anything else | Application failure | 500 |

### What each protocol delivers to the endpoint

| Client sends | Translator input | Neuron receives |
|---|---|---|
| AG-UI tool message `content` (string) | `AGUIInputTranslator` | Text result; JSON stays text |
| AG-UI tool message with `error` string | `AGUIInputTranslator` | Error result |
| AG-UI `resume[]` `{approved, reason?}` for a `confirmation` interrupt | `AGUIInputTranslator` | Approve / reject the action |
| Vercel part `output-available` with any `output` | `VercelAIInputTranslator` | Objects and arrays as JSON text, `false`/`0`/`null` as `"false"`/`"0"`/`"null"` |
| Vercel part `output-error` with `errorText` | `VercelAIInputTranslator` | Error result |
| Vercel part `approval-responded` `{approval: {id, approved, reason?}}` | `VercelAIInputTranslator` | Approve / reject the action |
| AG-UI `resume[]` resolved with a result map, for the reloaded frontend wait | `AGUIInputTranslator` | The results of the pending calls |
| AG-UI `resume[]` cancelled, for the reloaded frontend wait | `AGUIInputTranslator` | An error result for every pending call |

### Reloading an AG-UI conversation

A reloaded page rebuilds its client from storage. `AGUIAdapter::hydrate()` returns what the client held when the live stream ended: `messages` for `initialMessages`, `interrupts` for `pendingInterrupts`.

```php
use NeuronAI\Agent\Adapters\AGUIAdapter;
use NeuronAI\Workflow\WorkflowEngine;

// GET /agui/threads/{threadId}, after authorizing the thread
$run = (new WorkflowEngine($persistence))->inspect($threadId);
$body = (new AGUIAdapter($threadId))->hydrate($messageStore->loadAll($threadId, limit: 50), $run);
```

- Messages Neuron wrote keep the IDs the client rendered live; messages the browser created (the user's, frontend tool results) come back with their stored IDs.
- A pending approval comes back as the same `confirmation` interrupts the live stream sent.
- Frontend calls still waiting for results come back with one interrupt, reason `neuron:wait_for_event`, because the reload discarded the browser execution the live stream had started. Resume it `resolved` with Neuron's result map, or `cancelled` to settle every pending call as an error. Results already delivered come back as tool messages.
- A running or failed run returns no interrupts, and calls without results stay hidden until they have one. The question of a turn that has no answer yet still shows.
- Pass the run only with the latest page. Load older pages with `hydrate($olderPage, null)`, using the stored ID of the first loaded message as `loadAll()`'s `before`, never an AG-UI ID such as `result_…`.
- A reload does not reattach to a stream still running: show the working state from `$run->status` and hydrate again when the run settles.

## Vercel AI SDK (`useChat`)

Tested with `ai` 7.x and `@ai-sdk/react` 4.x.

```tsx
import { useChat } from "@ai-sdk/react";
import { DefaultChatTransport, getToolName, isToolUIPart } from "ai";
import type { UIMessage } from "ai";

function lastStepTools(messages: UIMessage[]) {
  const last = messages.at(-1);
  if (last?.role !== "assistant") return [];
  const step = last.parts.reduce((index, part, i) => (part.type === "step-start" ? i : index), -1);
  return last.parts.slice(step + 1).filter(isToolUIPart);
}

// Continue automatically after approvals and after every tool of the step has an outcome.
function lastStepIsSettled({ messages }: { messages: UIMessage[] }): boolean {
  const tools = lastStepTools(messages);
  if (!tools.length || tools.some((part) => part.state === "approval-requested")) return false;
  if (tools.some((part) => part.state === "approval-responded")) return true;
  return tools.every((part) => ["output-available", "output-error", "output-denied"].includes(part.state));
}

const { messages, sendMessage, addToolOutput, addToolApprovalResponse, status, error } = useChat({
  id: threadId,
  transport: new DefaultChatTransport({ api: "/api/chat" }),
  sendAutomaticallyWhen: lastStepIsSettled,
  // useChat awaits onToolCall: start the work and return, never await it here.
  onToolCall({ toolCall }) {
    const { toolName, toolCallId } = toolCall;
    void executeInBrowser(toolName, toolCall.input as Record<string, unknown>)
      .then((output) => addToolOutput({ tool: toolName, toolCallId, output }))
      .catch((failure) => addToolOutput({ tool: toolName, toolCallId, state: "output-error", errorText: (failure as Error).message }));
  },
});
```

Approval UI on a part in state `approval-requested`:

```tsx
<button onClick={() => addToolApprovalResponse({ id: part.approval.id, approved: true })}>Approve</button>
<button onClick={() => addToolApprovalResponse({ id: part.approval.id, approved: false, reason: "Not now" })}>Reject</button>
```

Part states you will see, in order: `input-streaming` → `input-available` (handler runs) → `output-available` | `output-error`; with approval: `approval-requested` → `approval-responded` → then the execution states, or `output-denied` after a rejection. A backend tool executed on the server appears as a part that goes straight to `output-available`; the browser handler is not invoked for it.

Rules the suite enforces:

- **Never await work inside `onToolCall`.** The SDK awaits the callback, so an awaited slow tool blocks stream processing and no other output is submitted until it finishes.
- **Predicates must be step-scoped.** After the final answer the earlier tool parts still exist on the assistant message. A predicate that looks at the whole message resubmits them and the endpoint answers 400 (no persisted run).
- **An eager predicate must remember what it sent.** If you submit as soon as any output is ready, a partial answer from the backend leaves the client state unchanged, so "any output available" resubmits the same batch after every response. Track submitted call IDs. The SDK's `lastAssistantMessageIsCompleteWithToolCalls` waits for all outputs and does not have this problem, but it does not cover mixed approval batches.
- **Reload.** `useChat` state is in memory. Persist `messages` yourself (the suite uses session storage keyed by thread) and pass them back as `messages` on mount; an `approval-requested` part restored this way can be answered normally.
- **Errors.** Before headers, `status` becomes `error` and `error.message` carries the HTTP body. After headers, the `error` part carries the protocol error text.

## AG-UI official client (`@ag-ui/client`)

Tested with `@ag-ui/client` 1.0.x. The protocol is what CopilotKit speaks underneath; use the client directly for contract tests or a custom frontend.

```ts
import { HttpAgent } from "@ag-ui/client";
import type { Interrupt, Tool } from "@ag-ui/client";

const tools: Tool[] = [
  { name: "read_title", description: "Read the title of the page the user is looking at.", parameters: { type: "object", properties: {} } },
];

const agent = new HttpAgent({ url: "https://app.example/agui", threadId });
agent.addMessage({ id: "user-1", role: "user", content: "What is the title?" });

const calls: Array<{ id: string; name: string; args: Record<string, unknown> }> = [];
let interrupts: Interrupt[] = [];
await agent.runAgent({ tools }, {
  onToolCallEndEvent: ({ event, toolCallName, toolCallArgs }) => calls.push({ id: event.toolCallId, name: toolCallName, args: toolCallArgs }),
  onRunFinishedEvent: (params) => { if (params.outcome === "interrupt") interrupts = params.interrupts; },
  onRunErrorEvent: ({ event }) => console.error(event.message),
});
```

Answering calls and approvals:

```ts
// A frontend executes only the calls it owns that carry no result yet:
// backend-executed calls arrive together with their TOOL_CALL_RESULT.
const answered = new Set(agent.messages.filter((m) => m.role === "tool").map((m) => m.toolCallId));
for (const call of calls.filter((c) => owned.has(c.name) && !answered.has(c.id))) {
  try {
    const value = await execute(call);
    agent.addMessage({ id: `result-${call.id}`, role: "tool", toolCallId: call.id, content: typeof value === "string" ? value : JSON.stringify(value) });
  } catch (error) {
    agent.addMessage({ id: `result-${call.id}`, role: "tool", toolCallId: call.id, content: "", error: (error as Error).message });
  }
}
await agent.runAgent({ tools });

// An approval is a `confirmation` interrupt; resume must address every open interrupt.
await agent.runAgent({ tools, resume: [{ interruptId: interrupts[0].id, status: "resolved", payload: { approved: true } }] });
```

Facts the suite established:

- An ordinary frontend handoff ends with `RUN_FINISHED` without an outcome; the tool messages on the next request continue it. Only approvals and other waits produce `outcome.type === "interrupt"`.
- Tool message content is text. Send structured values as JSON text; Neuron passes the text through unchanged.
- `runAgent` **resolves** on `RUN_ERROR` and reports it through `onRunErrorEvent`; it rejects only on HTTP failures before the stream (the error message carries the status and JSON body).
- A failed result streams its error text as the `content` of `TOOL_CALL_RESULT`. AG-UI defines `error` on the tool message, not on that event, so the frame has no `error` field and `agent.messages` built from the stream never carries one; the marker appears only on tool messages in a `MESSAGES_SNAPSHOT` or a reload. A rejected approval is a plain instruction string with no `error` marker at all.
- Two calls of the same tool in one batch stay distinct by call ID across requests.
- **Reload.** Seed a fresh client from the reload endpoint. `runAgent` refuses to run while a pending interrupt is not addressed by `resume`, so the reloaded page answers the persisted approval or frontend wait before anything else. After the run completes, another reload rebuilds exactly the message IDs the client holds.

```ts
const { messages, interrupts } = await (await fetch(`/agui/threads/${threadId}`)).json();
const agent = new HttpAgent({ url: "https://app.example/agui", threadId, initialMessages: messages });
agent.pendingInterrupts = interrupts;

// A frontend wait restored by the reload: deliver the results, or cancel the wait.
await agent.runAgent({ tools, resume: [{ interruptId: interrupts[0].id, status: "resolved", payload: { call_1: { result: "Page title" } } }] });
```

## CopilotKit

Tested with `@copilotkit/react-core` and `@copilotkit/runtime` 1.x using the v2 API. CopilotKit's hooks talk to a CopilotKit runtime, which drives the Neuron AG-UI endpoint through `HttpAgent`. That bridge is a small Node process you own:

```js
import { createServer } from "node:http";
import { HttpAgent } from "@ag-ui/client";
import { CopilotRuntime, createCopilotRuntimeHandler } from "@copilotkit/runtime/v2";
import { createCopilotNodeHandler } from "@copilotkit/runtime/v2/node";

const handler = createCopilotNodeHandler(
  createCopilotRuntimeHandler({
    runtime: new CopilotRuntime({ agents: { default: new HttpAgent({ url: "http://127.0.0.1:8787/agui" }) } }),
    basePath: "/runtime",
  }),
);
createServer((request, response) => void handler(request, response)).listen(4000);
```

Do not mount the runtime under the same path as a page of your app: a proxy rule for `/copilotkit` also captures a page served at `/copilotkit/`.

```tsx
import { CopilotKit, CopilotChat, useFrontendTool, useInterrupt } from "@copilotkit/react-core/v2";
import { z } from "zod";
import "@copilotkit/react-core/v2/styles.css";

function ReadTitleTool() {
  useFrontendTool({
    name: "read_title",
    description: "Read the title of the page the user is looking at.",
    parameters: z.object({}),
    handler: async (_args, context) => {
      // context.toolCall.id is the call ID Neuron will see the result under
      return document.title;
    },
  });
  return null;
}

function Approvals() {
  useInterrupt({
    render: ({ interrupt, resolve }) => (
      <div>
        <button onClick={() => resolve({ approved: true })}>Approve</button>
        <button onClick={() => resolve({ approved: false, reason: "Not now" })}>Reject</button>
      </div>
    ),
  });
  return null;
}

<CopilotKit runtimeUrl="/runtime">
  <ReadTitleTool />
  <Approvals />
  <CopilotChat threadId={threadId} />
</CopilotKit>
```

Facts the suite established:

- **Registration follows component lifecycle.** A tool whose component is not mounted is absent from `RunAgentInput.tools`; if the model still calls it, the run fails with `RUN_ERROR` and is recorded as failed. Mount tool components for the whole conversation.
- **Follow-up is automatic** after handlers return: CopilotKit runs the agent again with all results, once per step, and stops after a final answer.
- **No error channel.** A throwing handler becomes a tool message whose content starts with `Error:`, and CopilotKit starts **no** follow-up run. The Neuron run stays suspended until the application acts (for example by catching in the handler and returning a result the model can read).
- **Result normalization.** Objects and arrays are JSON text, `false` and `0` are their text, `null`/`undefined` become an empty string.
- **Reload.** CopilotKit restores the thread, including an open approval interrupt, from the runtime bridge's in-memory thread store, as long as the bridge process is alive and `threadId` is stable.
- **Partial submission** is not available: handlers are awaited and results are submitted together.

## Retries, Identity, Restart

| Situation | Outcome |
|---|---|
| Same partial submission delivered again while the batch is pending | Accepted, no extra inference |
| Different outcome for an already settled call | `WorkflowException`, nothing executes (endpoint: 409) |
| Submission after the run completed | `InputTranslationException` "no persisted run" (400), never a new turn |
| Submission carrying results of a previous turn | `InputTranslationException` "no matching continuation input" (400) |
| Backend process restarted between requests | A fresh process on the same database continues the wait; backend tools already executed are not rerun |
| Provider fails after streaming started | Run recorded as `failed`, no wait left behind, client sees `RUN_ERROR` / `error` part |

Neuron identifies results by call ID inside the active run only. It does not know whether a browser side effect already happened before its result was lost; that retry policy, and any deadline for a wait, belongs to the application. Cancelling an arbitrary wait and approving with edited arguments are not supported by the built-in translators.

## Version Compatibility

Each client stack is supported on its current major line; the integration suite is the compatibility statement and its lockfile records the last verified versions. A new major of any client requires a review and a new suite run.

| Stack | Supported line | Last verified |
|---|---|---|
| AG-UI (`@ag-ui/client`, `@ag-ui/core`) | 0.0.x | 0.0.59 |
| Vercel AI SDK (`ai`, `@ai-sdk/react`) | 7.x / 4.x | 7.0.98 / 4.0.101 |
| CopilotKit (`@copilotkit/react-core`, `@copilotkit/runtime`, v2 API) | 1.x | 1.71.0 |
| React | 19.x | 19.3.0 |

## Related

- **neuron-tool-approval** — the approval gate, decision translators and history-based approval UIs.
- **neuron-streaming** — the outbound half of this round trip: stream adapters, terminal frames on suspension, portable progress events, custom adapters and push channels.
- **neuron-tool** — writing backend tools and toolkits.
- **neuron-laravel-integration**, **neuron-symfony-integration** — these endpoints as framework controllers, with thread authorization, error mapping, reload and queued runs.
- `tests/Integration/Frontend/README.md` — the suite layout, scenarios and the full list of verified behaviours.