The in-process middleware chain — RPC-edge interceptors and whose traceparent the edge adopts (observe.TrustRemoteBy), the generated middleware_gen.go component decorator, the owned observe_chain.go seam, success-log sampling, and opting out.
Installs into .claude/skills of the current project.
Sponsors
SSkills DirectorySkills Directory
Know which skills are safe — weekly.
SSkills DirectorySkills Directory
Is that skill safe to install?
Are you the author of Component Chain?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/reliant-labs-component-chain)
---
name: observability/component-chain
description: The in-process middleware chain — RPC-edge interceptors and whose traceparent the edge adopts (observe.TrustRemoteBy), the generated middleware_gen.go component decorator, the owned observe_chain.go seam, success-log sampling, and opting out.
---
# Observability: the component middleware chain
Forge instruments three boundaries, so a request is observable end to end:
1. **The RPC edge** — every Connect handler runs the interceptor chain built by
`serverkit.NewChain(serverkit.ChainDeps{…})` in `cmd/<bin>/cmd/serve.go`
(otelconnect → recovery → request-id → logging → tracing → metrics, then
deadline → auth → audit → rate-limit → activity → stream-drain →
protovalidate → svcerr, which `observe.Chain` appends itself). One
span, one metric sample and one log record per RPC:
`rpc failed` for every failure, `rpc completed` for every success, and
`slow=true` on any success over 1s. Success logging can be SAMPLED — see
[Success-log sampling](#success-log-sampling) below. Tune the layer
through `serverkit.ChainDeps.LogOptions` —
`observe.WithSuccessLevel(<svc>connect.<Svc><Method>Procedure, slog.LevelDebug)`
silences one poll, `observe.WithSuccessSampling(d)` fixes this layer's
sampling window, `observe.WithSlowThreshold(d)` moves the slow line.
Those lines are written when an RPC COMPLETES; the chain's Deadline slot
(`serverkit.Deadline`) also watches a unary RPC while it runs, logging
`rpc still in flight` at WARN after `SlowAfter` (10s) and again every
`SlowRepeat` (30s), then `rpc completed slowly` — so a wedged handler shows
up in the log tail before its deadline (30s by default; per rpc from
`(forge.v1.method).timeout`) ends it. `Deadline.OnSlow` replaces that log,
e.g. to count slow calls in a metric too. When that deadline ends the
call, `rpc failed` is a WARN with `deadline=server` (an ERROR under
`Deadline.ExpiryIsError`); a caller's shorter deadline is WARN with
`deadline=caller` (see `observability`, "Whose deadline").
2. **The in-process component boundary** — every internal component→component
method call (a `contract.go` `Service`) gets one span, one metric sample
and one log record (every failure, every success unless sampling is on),
plus panic-recovery. This is the layer detailed below — the in-process
twin of the edge chain.
3. **The ORM** — `pkg/orm` registers the bun `bunotel` query hook, so every DB
query becomes a child span.
The middle layer used to be dark. forge **no longer** scaffolds a hand-written
`observe.go` / `NewObserved` decorator that you extend by hand. Instead it
generates a slim per-method decorator — `middleware_gen.go` (forge-owned,
`// Code generated by forge. DO NOT EDIT.`, regenerated from the `Service`
interface on every `forge generate` and hash-tracked exactly like its sibling
`mock_gen.go`). Each generated wrapper method is a one-liner routing the inner
call through a `*observe.ComponentChain` (via `chain.Around` for the
`(T, error)` shape, `chain.Run` for the rest), so adding a method to the
interface regenerates the wrapper — there is nothing to maintain by hand. The
exported constructor is `New<Concrete>WithForgeMiddleware(inner Service)
Service`, named after the constructor's concrete return type: the canonical
`func New() Service { return &service{} }` yields `NewServiceWithForgeMiddleware`,
and the composition site emits `pkg.NewServiceWithForgeMiddleware(pkg.New(...))`
around your untouched `New`.
### Whose `traceparent` the edge adopts
It is the chain's Trace slot's choice (`serverkit.ChainDeps.Trace` / `observe.Deps.Trace`). The default adopts every caller's — right when the browser is first-party. A server reachable by callers who must not choose its trace ids uses `observe.TrustRemoteBy(classify)`, whose classifier authenticates each caller from its headers and grants one of three levels: `TrustAll` (an internal service: ids, sampled flag, tracestate, baggage), `TrustIDs` (a signed-in end user: the trace and parent span ids only — the server makes its own sampling decision from `OTEL_TRACES_SAMPLER`, and tracestate and baggage are dropped), or `TrustNone` (a new trace linked to the caller's). `observe.TrustRemoteFrom(predicate)` is the two-level form. The threat model is in `pkg/observe/trust_remote.go`.
### The owned seam: `observe_chain.go`
You never touch the generated decorator. The chain it routes through is
assembled in an OWNED, scaffold-once file next to `contract.go`:
```go
// observe_chain.go — yours: scaffolded once, never overwritten by forge.
func newObserveChain() *observe.ComponentChain {
scope := "<module>/internal/<pkg>"
logger := slog.Default()
return observe.NewComponentChain(
observe.RecoverMiddleware(logger), // panic -> error, logged with stack
observe.TraceMiddleware(otel.Tracer(scope)), // one span "<pkg>.<Method>" per call
observe.MetricsMiddleware(otel.Meter(scope), "<pkg>"), // <pkg>.calls / .errors / .duration
observe.LogMiddleware(logger, slog.LevelDebug), // every failure; every success (or a sample)
)
}
```
Middlewares run outer→inner in the order listed; each is nil-safe (no configured
tracer/meter degrades to pass-through), so a decorator wired in a test harness is
always safe. This file is THE extension point:
- **Add a layer** — implement `observe.ComponentMiddleware`
(`WrapComponent(ctx, method, next) error`) and append it (an in-process
timeout or rate-limit layer, say).
- **Drop a layer** — delete its line (a nil entry is dropped too).
- **Change the success-log level** — the `observe.LogMiddleware` argument.
Failures log at their class's level (ERROR for a server fault, INFO for a
user error or cancellation — see
[Error levels](#error-levels-who-has-to-act)); successes log at this
level. The scaffolded
default is seeded from `observability.log_level` in forge.yaml (`debug` |
`info` | `warn` | `error`; default `debug`, so success stays quiet under a
production Info handler).
- **Tune success logging** — every success is logged (see
[Success-log sampling](#success-log-sampling)); a call over 1s is always
logged with `slow=true`. Trailing options on `observe.LogMiddleware` change
that for this package:
`observe.WithSuccessLevel("<pkg>.<Method>", slog.LevelInfo)` for one method,
`observe.WithSuccessSampling(d)` to sample this package's successes
(`0` = every success),
`observe.WithSlowThreshold(d)` to move the slow line.
- **Declare expected errors** — an error that is an ANSWER, not a failure
(a storage read for a missing object behind a 404, a registry asked for a
repository it does not hold), is declared on `observe.LogMiddleware`:
`observe.WithExpectedErrors(ErrNotFound)` (`errors.Is` against each
target), or `observe.WithExpectedErrorFunc(func(err error) bool)` for a
classification no sentinel expresses. A declared error is logged like a
success — at the success level, sampled in its own slot, carrying the
error and `expected=true` — so it is quiet under a production INFO handler
while every undeclared error keeps its class's level. Nothing is expected
until declared. The same options work at the RPC edge through
`observe.Deps.LogOptions` (a declared NotFound logs `rpc failed` at the
procedure's success level, sampled with its successes). Logging only: the
span and `<pkg>.errors` still record the error.
- **Change the user-error level** — `observe.WithUserErrorLevel(slog.LevelWarn)`
on `observe.LogMiddleware` (or in `observe.Deps.LogOptions` for the RPC
edge) for a deployment that wants user errors in a WARN view.
The chain captures only method identity, duration, and error status — never
arguments or results. It records `<pkg>.calls` / `<pkg>.errors` /
`<pkg>.duration` (each tagged `method="<pkg>.<Method>"`) and one span named
`<pkg>.<Method>` per call.
### Success-log sampling
Every successful call is logged by default — at the RPC edge and at every
component boundary. That is what you want in dev, and in any process whose
traffic you can afford to read. Where an access log's volume (which is the
traffic's volume) outgrows its value — a frontend polling three RPCs every
couple of seconds wrote 47k `rpc completed` lines in twelve hours of one dev
stack — turn on sampling for that deployment:
```kcl
# deploy/kcl/<env>/config.k
app_config: config_gen.AppConfig = {
log_success_sample_window = "1m"
}
```
`log_success_sample_window` is declared in `proto/<project>/config/v1/config.proto`,
so the config loader types and validates it, and it can come from any source
the loader reads (`--log-success-sample-window`, `LOG_SUCCESS_SAMPLE_WINDOW`,
a `--config` file). The scaffolded `serve.go` passes the loaded value to the
RPC edge's logging interceptor:
```go
chain, err := serverkit.NewChain(serverkit.ChainDeps{
// ...
LogOptions: []observe.LogOption{observe.WithSuccessSampling(cfg.LogSuccessSampleWindow.AsDuration())},
})
```
forge's `obs` package reads no environment — the window is only ever
the one its caller passed. So an older project adopts it by adding the field
to its AppConfig and that one `LogOptions` line to its owned `serve.go`; a
server that is not forge-scaffolded passes `observe.WithSuccessSampling` in
`DefaultMiddlewareDeps.LogOptions` from its own config.
Sampled, each RPC procedure logs its first success, then at most one per
window carrying `suppressed=<n>` — the successes that record stands for, so
the rate survives in the log. Volume is bounded by the number of procedures,
not by traffic. Never sampled, at any setting: failures (every one, with its
error) and successes slower than 1s (`slow=true`).
Component-call logs (`observe_chain.go`) default to DEBUG, so a production
INFO handler already drops their successes. To sample one package's anyway,
append `observe.WithSuccessSampling(d)` to its `observe.LogMiddleware`.
`0` (or a negative window) logs every success. When a layer is given more
than one `WithSuccessSampling`, the last one wins.
### On by default; opting out
- **Every contract package is instrumented** — a package whose constructor
returns its `Service` interface gets `middleware_gen.go`, and compose.go
constructs it wrapped, with no marker and no seam. Without
`observe_chain.go` the decorator builds `observe.DefaultComponentChain`
(recovery, span, metrics, log at `observability.log_level`); write the seam
to change that one package. `forge generate` also rewraps (or unwraps) a
construction compose.go already has when the decision changes.
- **Opt a package out** — `// forge:no-observe` on the constructor (or the
package / contract-interface doc). No decorator is generated and the
composition site falls back to the unwrapped `pkg.New(...)`. Do it for a
hot, trivial package (a pure function called in a loop), not to save spans
in general: a trace with no business-method spans cannot say where a
request spent its time.
- **Opt ONE method out** — `// forge:no-observe` on that interface method's doc
comment. The decorator still satisfies the interface, but that method
delegates straight to the inner impl, around the chain.
- **Handler packages are never wrapped** — they return a concrete `*Service`
and otelconnect already owns the RPC edge; wrapping would change the
`Components` field type.
- **Constructions you write by hand** (in `providers.go`, or a `*_wire.go`)
are yours: wrap them yourself —
`infra.X = pkg.NewServiceWithForgeMiddleware(pkg.New(...))`.
For a one-off child span or metric at a single call site (rather than a whole
decorator), `observe.LogCall` / `observe.TraceCall` / `observe.NewCallMetrics`
remain available.