Skill

---
name: failproofai-sdk
description: |-
  The way to make an AI agent report what it did to Failproof AI — planning what to record, writing the instrumentation, and proving the events land. Reach for it on vague phrasing too: "add observability to my agent", "why isn't my agent showing up?"

  Trigger when the user wants to:
  • plan an integration — which points in their agent loop to record, and what the platform must see before sessions, errors, and evals work at all;
  • write or fix instrumentation — add the `failproofai_sdk` Python SDK to an agent codebase, thread session/agent identity through it, emit tool, model, hook, or human events;
  • verify it — confirm events are being written, or debug an integration that looks correct and produces nothing.

  Served by the `failproofai_sdk` Python SDK, inside the user's own agent.

  NOT for reading telemetry that already landed or operating a deployment (that's `fp-cloud-cli`), or building the evaluator service that scores runs (that's `agenteye-evaluator`).
---

# Failproof AI Python SDK

The SDK records what your agent did, from inside your agent. You call it at points
you choose; it appends structured events to local `.jsonl` files. A separate
collector ships those files to the platform.

```
your agent calls failproofai_sdk.event.*
  → SDK queues it in memory
  → flush thread writes <base_dir>/events/event-<timestamp>.jsonl
  → collector picks the file up and ships it
  → visible as sessions / events / errors / evals
```

**The SDK's job ends at the file.** That boundary is the most useful thing to know
about it: everything up to the `.jsonl` is yours to get right and yours to verify,
and it is verifiable on a laptop with no server, no API key, and no network.

The API is small — 15 event methods, all keyword-only. The hard parts are
**deciding where to call them** and **knowing which silences are bugs**, because
this SDK does not raise when you get it wrong. Sections 1-3 are the plan, 4 is the
code, 5-6 are the proof.

## 1. Install it

```bash
pip install failproofai-sdk        # or: uv add failproofai-sdk
```

The distribution is `failproofai-sdk` and the import is `failproofai_sdk`. Public
PyPI, no token, no dependencies.

**One command to never run: `pip install agenteye`.** That name belongs to a
stranded release of an old CLI — a different product that shipped under it before
moving to `fp-cloud-cli`. PyPI versions cannot be withdrawn, so the name still resolves
to that build forever. You get the CLI, `import failproofai_sdk` raises
`ModuleNotFoundError`, and on a codebase still using the pre-rename SDK (which
published under `agenteye` too) pip treats it as an upgrade and **removes the SDK**.

> **Tell:** if a coding agent proposes `pip install agenteye` to install the SDK,
> this skill never loaded. Stop and re-read it.

The CLI is a fine thing to want — it is what reads the telemetry back. Install it
separately, never with `pip` into your agent's environment:

```bash
pipx install fp-cloud-cli        # the command is `fp`
```

Confirm what you actually have before writing a line of instrumentation:

```bash
python -c "import failproofai_sdk; print(failproofai_sdk.__version__)"
```

A version like `0.0.1b1` is the SDK. `ModuleNotFoundError` means it is not
installed — check `pip show agenteye`, which returning anything means the wrong
name was installed. `references/install.md` covers migrating an existing
`import agenteye` integration.

## 2. Plan before you instrument

Instrumentation lands in code that already exists and already works. Read it
first, then decide. Two questions settle most of the design, and only the user can
answer the first:

> **What is one run of this agent?** That is your `session_id` — one value for the
> whole run, generated by you at the point the run starts. A chat turn, a job, a
> request, a workflow execution. If the agent handles concurrent runs, this must
> be per-run, not per-process.
>
> **What are the distinguishable actors in a run?** That is your `agent_id` — a
> stable *label*, not a unique id. `"planner"`, `"researcher"`, `"main"`. It is how
> the platform tells sub-agents apart, so reuse the same string across runs.

Get these two named and agreed before writing code. They are the axes every
surface groups by, and changing them later splits the history: old runs keep the
old labels and the trends break.

### The two events everything else hangs off

Most of the catalog is optional and incremental. These two are not:

| Event | Without it |
|---|---|
| `agent_start` | **The session does not exist.** No row on Sessions, no timeline, no evaluation — while every other event you emit still lands fine and shows up in the event stream. |
| `agent_end` | The run never closes, and it is not handed to the evaluator at the normal time. |

That first row is the single most common integration failure, and it is
completely silent: a run emitting 500 tool calls and no `agent_start` produces a
busy event stream and **zero sessions**. Sessions are *defined* as "something that
emitted `agent_start`". So:

**Emit `agent_start` at the top of the run and `agent_end` at every exit, and get
those two working end-to-end before you instrument anything else.** One event at
each end proves the whole path — install, identity, base dir, collector — with
almost no code to be wrong. Add tools, models, and hooks after that path is green.

### Then map the rest onto the agent's shape

Walk the agent loop and pick the points that exist in *this* codebase. Skip what
doesn't apply; there is no requirement to emit every type.

| In the code | Emit | Buys you |
|---|---|---|
| every exit path of a run — success, exception, early return | `agent_start` / `agent_end` | the session itself |
| the tool dispatcher, both sides of the call | `tool_use` / `tool_result` | what ran, in what order, how long |
| the LLM client wrapper, both sides | `model_request` / `model_response` | model mix, token spend, stop reasons |
| your `except` blocks | `error` | the Errors surface |
| a policy/guard/middleware layer | `hook_triggered` / `hook_completed` | hook behaviour |
| an approval gate or human handoff | `human_wait` / `human_input`, `human_pause`, `human_interrupt` | where runs sit waiting on people |
| a run that suspends and resumes — waiting for a human, throttled, user-paused | `agent_pause` / `agent_resume` | a real "paused" state: the agent isn't ended, the resume isn't a new agent, and wait time is excluded from active work |

If the codebase has one tool dispatcher and one LLM wrapper, you have two edit
sites for the bulk of the value. If tool calls are scattered inline across the
codebase, say so — a wrapper (§4) is worth more than 40 call sites.

Full field-by-field catalog: `references/events.md`.

## 3. The contract

Work with these; none of them raise, so none of them show up in testing.

- **There IS an ambient session, and it is the ergonomic path.** `session()`,
  `agent()` and `tool_call()` bind identity on contextvars, so `session_id` and
  `agent_id` are optional on all 15 event methods — omitted, they resolve from
  the enclosing scope. `current()` reads it; `propagate(fn)` carries it into a
  new thread, which contextvars do NOT do on their own.

  This section said the opposite until the scopes existed, and the reference
  integration shipped a contextvars wrapper as markdown for customers to paste
  into their own code. That is now in the package.

  Nothing bound and nothing passed raises `TypeError` naming the fix — never a
  silent emit, because ingest skips an event with no session and answers `200`.

  Two more shapes raise, for the same reason:

  | You pass | Raises | Why it cannot be allowed through |
  | --- | --- | --- |
  | A non-`str` id | `TypeError` | Ingest skips the event and still answers `200` |
  | `""` or `"   "` | `ValueError` | Worse — ingest *accepts* it, and every event merges under one blank id |

- **`configure()` is optional, and every call restates all of it.** It is
  keyword-only with exactly three settings:

  | arg | default resolution |
  |---|---|
  | `base_dir` | `~/.failproofai/custom-agents` (honours `$FAILPROOFAI_HOME`) |
  | `environment` | `$AGENTEYE_ENVIRONMENT`, else `"dev"` |
  | `flush_interval` | `0.5` (seconds) |

  **No environment variable can move the spool out of the umbrella.**
  `$FAILPROOFAI_HOME` relocates the umbrella itself, but `custom-agents` is
  appended unconditionally, so the spool is always inside it. `base_dir` is the
  only way to write anywhere else, and it is an explicit argument at the call
  site rather than something inherited from the environment.

  The default root moved here from `~/.agenteye`. `failproofaid` watches both,
  so on a host running it nothing changes but the directory name, and batches
  already in `~/.agenteye/events` still get collected. On a host running the
  older `agenteye-collector` — which resolves `$AGENTEYE_HOME` or `~/.agenteye`
  and nothing else — point **the collector** at this SDK with
  `AGENTEYE_HOME=~/.failproofai/custom-agents`, or pass `base_dir` here.
  Setting `AGENTEYE_HOME` no longer moves the SDK: it used to, which meant
  exporting it for the collector silently relocated the SDK too.
  `AGENTEYE_SPOOL_TO_FAILPROOFAI` is retired; it required a directory nothing
  created, so it never fired.

  Each call *sets all three* — omitted arguments are **reset to default
  resolution**, not left alone. So a later `configure(flush_interval=1.0)`
  silently moves your events back to the default directory and re-resolves the
  environment. An explicit `configure(environment=...)` beats the env var; omit it
  and the env var applies again. Call it **once**, at startup, before the first
  event, passing every argument you care about.

- **`environment` defaults to `"dev"`.** An unconfigured production agent reports
  its runs as `dev` and they are invisible wherever the team filters on
  `production`. Set it explicitly via `configure(environment=...)` or the
  `AGENTEYE_ENVIRONMENT` env var. This is a favourite: everything works, in the
  wrong bucket.

  **A comma raises `ValueError`.** `configure(environment="prod,eu")` is rejected
  at the call site: ingest splits this field on commas to build filter facets, so
  a comma would discard the whole event server-side with nothing said.

- **Non-JSON payload leaves are stringified.** Events are serialized on a
  background thread. Ordinary structured JSON retains its types; unsupported
  leaves such as `datetime`, `UUID`, `Decimal`, `set`, `bytes`, or a Pydantic
  model are converted with `str(value)` so one awkward tool result cannot stop
  recording. Prefer plain JSON values when downstream queries need their
  structure; use explicit custom serialization when a string would be ambiguous.

- **Field *names* are unvalidated — but only the optional ones.** Every method
  takes arbitrary `**fields` and stores them as-is, so a typo'd *optional* name
  (`inpt=` for `input=`) is not an error, it is a new field, and nothing will tell
  you. Typos in *required* names raise `TypeError` (they're real parameters), and
  five reserved names — `timestamp`, `session_id`, `agent_id`, `type`,
  `environment` — raise `ValueError`.

- **`outcome="failed"`, not `"failure"`.** A run counts as failed only when
  `outcome` (or `status`) is one of `error`, `failed`, `timeout`, `rejected`
  (case-insensitive). `"failure"` is the natural antonym of the `"success"` in
  every example — and it silently counts as *not a failure*. The run shows green.

- **You own correlation, and ids are scoped per session AND per kind.**
  Pending spans are keyed `tool:<session_id>:<tool_call_id>` and
  `hook:<session_id>:<hook_id>`, so a `hook_completed(hook_id="x")` cannot pair
  with a pending `tool_use(tool_call_id="x")`, and two concurrent sessions both
  using `call_1` cannot cross-pair either. `input_id` and `pause_id` are scoped
  the same way.

  What must still be unique is an id **within one session, for one kind**. The
  pending map is a plain assignment, so emitting `tool_use(tool_call_id="call_1")`
  twice in one session overwrites the first entry and the first `tool_result`
  measures from the wrong start. Reusing your framework's id is always safe
  (Anthropic and OpenAI ids are globally unique); a per-run counter is safe only
  if you do not reset it inside a session.

  If you have read older guidance describing one flat, process-wide map shared
  between tools and hooks: that was true, and is not any more.

- **`duration_ms` is computed for you on four methods only** — `tool_result`,
  `hook_completed`, `human_input`, `agent_resume` — from the matching earlier event.
  Passing it to those four raises `ValueError`. Passing it to any of the other
  eleven is **silently accepted as a custom field**.

- **Events are fire-and-forget.** `event.*` queues in memory and returns; a daemon
  thread writes every 0.5s, plus once at interpreter exit. A clean exit flushes.
  A hard kill (`SIGKILL`, `os._exit`, a container OOM) drops whatever is queued,
  silently.

  **`SIGTERM` deserves its own line, because it is not exotic — it is every
  rolling deploy**, every `docker stop`, every Kubernetes eviction, and every
  plain `kill`. CPython installs **no** handler for it: `signal.getsignal(SIGTERM)`
  is `SIG_DFL`, the OS terminates the process where it stands, and **`atexit`
  does not run**. Whatever is queued is gone — and what is in flight at shutdown
  is disproportionately `agent_end`, so runs never close and never reach the
  evaluator. The 0.5s flush interval is what bounds the loss, not the exit path.

  So if your process can receive `SIGTERM`, handle it — the SDK will not install
  a handler in your process behind your back:

      import signal, sys, failproofai_sdk

      def _flush_and_exit(signum, frame):
          failproofai_sdk._writer.flush_now()
          sys.exit(128 + signum)

      signal.signal(signal.SIGTERM, _flush_and_exit)

  (`sys.exit` here rather than `os._exit`: it unwinds, so any `agent()` scope
  still open emits its `agent_end` before the flush. That scope closes
  `outcome="failed"` with an `error` naming `SystemExit`, because an evicted run
  did not finish — which is the thing you want to be able to see.) `SIGKILL`,
  `os._exit` and a container OOM cannot be handled by anything, and drop the
  queue silently.

## 4. Write it

Threading `session_id` and `agent_id` through every call site by hand is the thing
that makes integrations ugly and abandoned. Don't. Bind identity once per run and
let the call sites read it.

`references/frameworks.md` covers the four adapters. `references/integration.md` has the hand-written wrapper — one small
module, correct under `asyncio` and threads, adaptable to any codebase — plus
worked shapes for a tool dispatcher, an LLM client wrapper, and framework-specific
callback layers. Read it before writing your own; the naive version (a module
global, or a plain attribute) breaks the moment two runs overlap, and it breaks by
mixing two runs' events together rather than by failing.

Match the codebase you're in. If it's async, the wrapper is async. If it already
has a request context or a trace id, bind to that instead of inventing one.

## 5. Verify — watch the files

**This is the whole point of the file boundary: you can prove the integration
without a server.** Run the agent and look.

Resolve the spool the way the SDK does, rather than guessing at a path:

```bash
python -c "import failproofai_sdk._resolver as r; print(r.get_base_dir() / 'events')"
```

That prints `~/.failproofai/custom-agents/events` unless the application called
`configure(base_dir=...)`. `$FAILPROOFAI_HOME` moves the `~/.failproofai` part
and nothing else. `$AGENTEYE_HOME` does **not** affect it — that variable belongs
to the older `agenteye-collector`, which reads it to decide what to WATCH.

```bash
ls -la ~/.failproofai/custom-agents/events/
```

You are looking for `event-<UTC timestamp>-<pid>-<seq>.jsonl` files — the pid and
sequence number are what keep two processes flushing in the same millisecond from
overwriting each other. Each line is one event. Read them with a JSON parser, not
`grep` — the exact spacing is not a contract, and a grep for `"type":"agent_start"`
returns nothing on a perfectly healthy integration:

```bash
cat ~/.failproofai/custom-agents/events/*.jsonl | python -m json.tool --json-lines | head -20
```

Then check, in this order — the first failure explains everything downstream:

1. **Any files at all — or do they stop mid-run?** Look at stderr for
   `Exception in thread failproofai-sdk-flush`. **This is the first thing to check and
   the worst thing to miss**: one non-JSON-serializable value killed the writer,
   and everything after it — including the at-exit flush — is gone (§3). The tell
   is that events stop for *every* type at once, and nothing raised. If instead
   there were never any files: did `import failproofai_sdk` succeed (§1)? Is the base dir
   writable? Did the process die hard (`SIGKILL`, `docker stop`, an OOM) before a
   flush?
2. **Is `agent_start` there, once per run?** No → you will see events on the
   platform and no sessions, and you will spend an afternoon on it (§2).
3. **Sessions but no tool or model events?** Your emit path is dropping them
   before the SDK ever sees them — nearly always because they're emitted from a
   thread the identity never reached. See `references/integration.md` → *"Threads
   will drop your events"*. The SDK is silent here; only your own wrapper can warn.
4. **Is `environment` what you expect?** It is `"dev"` unless you set it (§3).
5. **Is `outcome` on `agent_end` a word that counts?** `failed`/`error`/`timeout`/
   `rejected` — not `"failure"` (§3). Failed runs showing green is this, every
   time.
6. **Run two overlapping runs.** Confirm two `session_id`s with **no events
   crossing between them**. Do not check this with one run: a single run passes
   even when identity is a module global, and mixing only appears once two runs
   overlap — which is production, not your laptop (§4).
7. **Do `tool_use` and `tool_result` share a `tool_call_id`?** Unpaired means no
   duration. Also confirm your ids are unique *process-wide* — a collision pairs
   the wrong two events and reports a confident wrong duration (§3).

A test-mode loop that costs nothing:

```bash
export FAILPROOFAI_HOME=/tmp/failproofai-sdk-test
rm -rf /tmp/failproofai-sdk-test && python your_agent.py
cat /tmp/failproofai-sdk-test/custom-agents/events/*.jsonl | python -m json.tool --json-lines
```

`FAILPROOFAI_HOME` sends events somewhere disposable, so you can iterate on the
integration without touching the real directory or shipping test runs to the
platform. Note the `custom-agents` segment in the read path — the SDK appends it
unconditionally. Note too that the SDK reads the variable late, per flush, so set
it before you start the process, not halfway through.

`AGENTEYE_HOME` used to do this job and no longer does anything to the SDK; using
it here would write to your REAL spool while you read an empty temp directory.

**Do not verify by installing the CLI into your agent's environment.** It will
uninstall the SDK you just integrated (§1). Reading back what landed on the
platform is the `fp-cloud-cli` skill's job, from a separate environment.

## 6. Production — the collector has to agree with you

The SDK writes files. It never talks to the network, so from its point of view a
completely unshipped integration looks perfect.

In production, the collector must be **running** and reading the **same directory
the SDK is writing to**. That is the whole contract, and both halves fail
silently:

- Collector not running → files pile up in `events/` forever. The SDK is fine.
- Collector reading a different base dir than the agent writes to — the
  collector's own `AGENTEYE_HOME` pointing somewhere else, a different user's
  `~`, a container path that isn't mounted → files pile up in a directory nobody
  reads. The SDK is fine. (`failproofaid` watches both
  `~/.failproofai/custom-agents/events` and `~/.agenteye/events`, so it is the
  half of this pair least likely to be misconfigured.)

So when events are on disk but not on the platform, the SDK is not the suspect.
Compare the two paths first: print the directory your agent is actually writing to
(`python -c "import failproofai_sdk._resolver as r; print(r.get_base_dir())"` in the
agent's own environment, with the agent's own env vars) and check the collector is
running and pointed at the same one. A `.jsonl` count that only grows is the tell.

Confirming events arrived on the *platform* is deliberately not this skill's job —
that is the `fp-cloud-cli` skill, from a **separate environment** (§1). Collector
setup and deployment are your platform's own documentation.

If the files look right (§5) and the collector is running against the same
directory, the integration is done.

<!-- ci: no-op touch to exercise the skill-sync trigger (safe to remove) -->