Kelly Agent Observability

---
name: kelly-agent-observability
description: Busabase App-in-Skill dashboard visualizing a generated MOCK fleet of LLM agents running behind a shared AI gateway for a generic organization. Use when the user invokes $kelly-agent-observability or /kelly-agent-observability, wants to review agent health, call volume, latency, error rates, cost, or trace-level chain breaks across an agent fleet. This is a demo/reference dashboard over a generated mock fleet snapshot seeded into Busabase by a trusted script — never a live monitoring integration, never a real gateway or real agents.
metadata:
  category: platform
  tags:
    - risk:sandbox
    - surface:busabase
  busabase:
    template: true
    folderSlug: kelly-agent-observability
    resources:
      - agents
      - traces
      - handoffs
      - settings
    risk: sandbox

---

# Agent Fleet Observability Desk

## Overview

Kelly Agent Observability is a Busabase Cloud App-in-Skill. Its canonical
product surface is the AirApp in Busabase, not a separate local-data
product. The same Hono source supports an explicitly requested local preview
with OAuth connection bootstrap. It gives an operator a read-mostly
dashboard over a MOCK fleet of LLM agents running behind a shared AI gateway
for a generic organization: per-agent call volume, latency (p50/p95),
error/failure rate, cost, an hourly call-volume sparkline, and a Trace Detail
view that highlights exactly where a chain of tool calls broke.

This skill was never a live monitoring integration, even before this
conversion — there is no real gateway, no real agents, and no real telemetry
anywhere. The fleet snapshot (agents + metrics + traces) is generated by a
deterministic, seeded function and written into Busabase by a trusted
skill-root script (`scripts/generate_fleet_data.mjs`); the AirApp itself only
ever **reads** whatever was last generated there. The only human action is
narrow: append an "acknowledge" / "needs investigation" handoff note to an
agent or trace — always a brand-new row in the `handoffs` Base, never a field
update on the agent/trace's own record.

Default behavior is AirApp-first. Unless the user explicitly asks only for
explanation, ensure the mock fleet exists (run the seed script below if the
`agents` Base is empty) and give the user the clickable AirApp URL. Start
localhost only when local preview/debugging is explicitly requested; it uses
the same Busabase resources and never offers another data provider. Use
chat-only mode only when the user says "纯聊天", "chat only", "no UI", or
similar.

This is deliberately **generic and brand-free**: agent names are generic
archetypes (Booking Assistant, Support Triage, etc.) for a generic
organization ("your organization", "the fleet") — no real company or product
appears anywhere in the code, config, or seed data.

## Mandatory Dependencies

1. Read and follow `$kelly-app-skill-creator` for product behavior, visual
   quality, responsive layout, and the complete canonical `content/kelly-agent-observability-app/` artifact.
2. Read and follow `$busabase` for connection, target Space, node discovery,
   ChangeRequests, review, and merge behavior.
3. Read and follow `$busabase-app-creator` for resource modeling, AirApp
   runtime limits, security, validation, and deployment.

If a dependency is unavailable, preserve this skill's artifact and product
contracts, stop before the unavailable Busabase operation, and report the
exact missing dependency. Do not invent a second data backend.

## App UI Screenshots

<table>
  <tr>
    <td width="50%"><img src="assets/screenshots/overview.webp" alt="Fleet overview"></td>
    <td width="50%"><img src="assets/screenshots/agent-health.webp" alt="Agent health table"></td>
  </tr>
  <tr>
    <td><strong>Fleet Overview</strong><br>Total calls, total cost, degraded/critical/healthy agent counts, and a per-agent sparkline card grid.</td>
    <td><strong>Agent Health</strong><br>Sortable table with call volume, p50/p95 latency, error rate, cost, and a status badge per agent.</td>
  </tr>
  <tr>
    <td width="50%"><img src="assets/screenshots/trace-detail.webp" alt="Trace detail with chain break"></td>
    <td width="50%"><img src="assets/screenshots/overview.zh-CN.webp" alt="Fleet overview in Chinese"></td>
  </tr>
  <tr>
    <td><strong>Trace Detail</strong><br>Ordered step timeline for one trace; the step where the chain broke is visually flagged.</td>
    <td><strong>Chinese UI</strong><br>Full zh-CN chrome via the language toggle or <code>lang=zh</code>.</td>
  </tr>
</table>

## Boundary

- Read-mostly dashboard over a generated mock fleet. NEVER contact any real
  agent gateway, LLM provider, or third-party API. There is no live
  monitoring integration in this skill by design.
- Agents and traces are never created or edited by the AirApp; they enter
  Busabase only through the trusted `scripts/generate_fleet_data.mjs` seed
  script, the same way `kelly-portfolio-health`'s contracts enter through an
  external sync process the app doesn't control.
- The only Base the AirApp itself ever writes to is `handoffs`, and only ever
  by creating a brand-new row (an acknowledge / needs-investigation note) —
  never a field update on an agent or trace record.
- Never reference any real company, brand, or production system. Agent names
  are generic archetypes for a generic organization ("your organization",
  "the fleet").

## Busabase Resources

Four Bases under one application Folder (`kelly-agent-observability`),
declared in `content/kelly-agent-observability-app/app/js/config.js` and the generated template sidecars under `content/`:

- `agents`: one row per mock agent archetype (8 rows) — identity plus the
  latest rolled-up health metrics (status, call volume, p50/p95 latency,
  error rate, cost) and a 48h hourly call/error series (JSON array). Written
  only by the trusted generator script.
- `traces`: one row per mock trace (ordered tool-call chain), capped to fit
  under the 100-record read limit (8 agents × 10 traces = 80 by default).
  `steps` (JSON array) carries the ordered timeline; `broke-at-step-id` flags
  exactly where a broken chain stopped. Written only by the trusted generator
  script.
- `handoffs`: append-only human-in-the-loop log — an acknowledge /
  needs-investigation note against one agent or trace. The only Base the
  AirApp itself ever writes to.
- `settings`: sanitized fleet-generation metadata (schema version, seed,
  generated-at), one row keyed by `kind`.

Resources provision lazily through an idempotent Busabase ChangeRequest the
first time the app runs in a Space; see `references/fleet-schema.md` for
exact field shapes.

## First Run And Onboarding

On invocation, check the `agents` Base. If it's empty, run the trusted seed
script to generate the mock fleet:

```bash
node skills/kelly-agent-observability/scripts/generate_fleet_data.mjs --apply
```

There are no credentials to collect — this skill never calls an external
system, so onboarding is just running the seed script once.

## Local App

Default behavior is AirApp-first — give the user the clickable AirApp URL.
Start `pnpm --dir content/kelly-agent-observability-app dev` only when local preview/debugging is explicitly
requested.

Required app views (hash routes):

- `#/overview`: total calls (24h), total cost (today), degraded/critical/
  healthy agent counts, and a per-agent card grid with a call-volume
  sparkline.
- `#/agents`: sortable Agent Health table (call volume, p50/p95 latency,
  error rate, cost, status badge).
- `#/agents/<agent_id>`: agent detail — metrics, sparkline, recent traces,
  and a handoff form (acknowledge / needs investigation).
- `#/traces/<trace_id>`: trace detail — ordered step timeline with the
  chain-break step visually flagged, and a handoff form.
- `#/handoffs`: history of recorded handoffs.
- `#/settings`: data provider and status-threshold summary.

## Demo Mode

- `?demo=1` opens a deterministic, fully offline mock fleet (8 agents, 48h of
  hourly buckets, 16 traces per agent) for documentation and screenshots. It
  never reads or writes Busabase and never claims a real connection; demo
  handoff submissions only update the in-memory list already rendered.
- `?demo=agents` and `?demo=trace` select named mock scenes (agent health
  table, a broken trace).
- `lang=en` or `lang=zh` forces UI chrome language for screenshots.

UI language: support English and Chinese chrome with `Auto` default.

## Fleet Generation Model

`content/kelly-agent-observability-app/app/js/fleet-model.js` (`generateFleetData`/`summarizeFleet`) is ported
verbatim from the retired `lib/generate.ts`:

- **Per-agent profile** — a fixed "personality" (volume, latency base/jitter,
  error tendency, cost per call) per agent archetype, so the fleet reads as
  varied, not random noise.
- **Hourly buckets** — 48 hours of calls/errors per agent on a
  business-hours-ish curve, seeded per agent.
- **Status** — `statusFor(error_rate_pct, p95_latency_ms)`: `critical` at
  error rate ≥ 8% or p95 ≥ 8000ms, `degraded` at ≥ 3% or ≥ 4000ms, else
  `healthy`.
- **Traces** — an ordered tool-call chain per trace; a trace can "break" mid
  chain (`status: error`, `broke_at_step_id` set to the last step), with the
  break probability tied to the agent's error tendency.

Same seed + same "now" + same `tracesPerAgent` always produces bit-identical
output — this is the property `scripts/generate_fleet_data.mjs` and the demo
provider both rely on.

## Handoffs

The one human action — acknowledge or flag "needs investigation" on an agent
or trace, with a free-text note — writes a brand-new row into the `handoffs`
Base via `busabase-sdk`, never a field update on the agent/trace record
itself:

- From a standalone local preview the write merges immediately (trusted
  operator).
- From the deployed AirApp it creates a pending ChangeRequest for the
  trusted process to merge, per the AirApp boundary in
  `$busabase-app-creator`.

## Safety

- Never invent a real incident or real company data; this is illustrative
  mock telemetry only.
- Keep all writes scoped to the `handoffs` Base. No external network calls
  anywhere in this skill.
- Redact nothing sensitive is collected here — there are no credentials in
  this skill.

## Useful Commands

```bash
node skills/kelly-agent-observability/scripts/generate_fleet_data.mjs --apply
pnpm --dir skills/kelly-agent-observability/content/kelly-agent-observability-app dev
```

## Execution reports

Re-read the active provider's decisions immediately before any approved execution. Record each concrete operation, target, status, timestamp, and error in the provider-backed execution report; keep app actions local-only.