Signals Scout Logs

---
name: signals-scout-logs
description: >
  Focused Signals scout for PostHog projects using logs. Watches for volume bursts,
  severity-distribution shifts, service silence, fresh message patterns, and
  trace-correlated bursts via the logs ingestion pipeline. Emits findings only when
  they clear the confidence bar; otherwise writes durable memory and closes out empty.
  Self-contained peer in the signals-scout-* fleet — no dependencies on other skills.
  Picked uniformly at random by the coordinator alongside `signals-scout-general` and
  other specialists.
compatibility: >
  Designed for the PostHog Signals agent in a Claude sandbox with PostHog MCP scopes (mostly read-only, plus
  signal_scout_internal:write for scratchpad-remember/forget and emit-signal). Assumes the signals-scout MCP family is available (project-profile-get, runs-list,
  scratchpad-search, scratchpad-remember, scratchpad-forget, emit-signal) plus
  the logs tool family (logs-count, logs-count-ranges, logs-sparkline-query, query-logs,
  logs-attributes-list, logs-attribute-values-list, logs-alerts-list).
metadata:
  owner_team: signals
  scope: logs
---

# Signals scout: logs

You are a focused logs scout. Spot meaningful changes in this team's log volume,
severity distribution, service activity, and fresh message patterns — and emit findings
only when they clear the confidence bar. Logs live in their own ingestion pipeline
distinct from `top_events`, so the project profile won't tell you whether logs are
loud today; you have to ask.

## Quick close-out: are logs even in use?

If `logs-count` over the last 24h returns zero or near-zero, this team isn't using
logs. Write one scratchpad entry:

- key: `not-in-use:logs:team{team_id}`
- content: brief note ("checked at {timestamp}, logs-count over 24h ≈ 0")

Close out empty. Future logs runs will read this entry cold and short-circuit in
seconds. Re-running with the same key idempotently refreshes the timestamp — the entry
stays until logs ingestion actually shows up, at which point the next run rewrites or
deletes it.

## How a run works

Cycle between these moves; skip what's not useful, revisit what is.

### Get oriented

Three cheap reads cold-start a run:

- `signals-scout-scratchpad-search` (`text=logs` or `text=service`) — durable team steering
  from past logs-focused runs. **Entries with `pattern:`, `noise:`, `addressed:`, or
  `dedupe:` key prefixes tell you what's normal, what's already surfaced, what to skip.**
- `signals-scout-runs-list` (last 7d) — what prior logs scouts found and ruled out.
- `logs-count` over 24h vs `logs-count` over 7d-prior 24h baseline — the cheap
  is-anything-loud-today check. `logs-count-ranges` adds severity / service breakdown.

### Explore

Patterns to watch — these are starting points, not a checklist.

#### Volume burst

`logs-count` over 24h is materially above the 7d-prior baseline (≥ 2x). Localize by
re-running `logs-count` (or `logs-count-ranges` for the time-bucketed shape) filtered
by `severity` and by `service` — these tools count a filter, they don't group, so
narrow with the filter and compare. Common causes: a stuck retry loop logging at
`info`, a feature deploy that bumped log verbosity, a misconfigured logger emitting
at `debug` in prod.

Cross-source convergence: if `top_events` shows `$exception` flat over the same window,
this is logs-exclusive — handled-but-real failures the application catches and logs but
doesn't re-raise. Distinct from anything error tracking will surface.

#### Severity distribution shift

Total volume flat but `error` / `fatal` proportion rising. Captures the kind of failure
error tracking misses: caught-and-logged exceptions, retry-with-eventual-success patterns,
degraded-but-functional dependencies (slow DB, cold cache, partial third-party outage).

Validate via `query-logs` filtered to `severity ∈ {error, fatal}` over the recent window,
grouped by `service` or `module`. A single service accounting for the rise is
high-confidence; a uniform rise across services suggests an upstream platform issue.

#### Service silence

A service that normally accounts for a meaningful share of total log volume drops to
near-zero. Different shape from error tracking entirely — there's no exception, the
service is just gone.

Validate: `logs-attribute-values-list` on `service` for active services, then
`logs-count-ranges` per service over today vs 7d-prior to confirm the missing service
was active before. Cross-check `top_events` for the service's expected user-facing
events — if those also dropped, the service is genuinely down.

#### Fresh message pattern

`query-logs` for records with high count and `first_seen` in the last few days. A
fresh message text repeated thousands of times indicates a new code path firing at
scale. Pull `logs-attributes-list` to see what structured fields the record carries
(`error_code`, `module`, stack-frame fields).

If the message references an exception, cross-check `query-error-tracking-issues-list` first
— if an issue already covers it, error tracking owns the finding.

#### Trace-correlated burst

Log records carrying `trace_id` correlating to slow or failing traces. When a
`query-llm-traces-list` failure spike, an `query-error-tracking-issues-list` burst, and a
`query-logs` burst all share the same trace ids — that's the cleanest cross-source
convergence pattern logs enables.

#### Alert without inbox coverage

`logs-alerts-list` exposes the team's configured alerts. An alert with `state =
firing` whose underlying condition isn't already in `inbox-reports-list` is a
high-confidence finding — the team has the alert plumbing but not the inbox surface.

### Save memory as you go

Memory is a continuous activity. Write a scratchpad entry whenever you observe something
a future logs run should know. Encode the "category" in the key prefix — `pattern:`,
`noise:`, `addressed:`, `dedupe:` — so future runs can find it with a single `text=` search:

- key `pattern:logs:temporal-worker` — _"Service `temporal-worker` typical log volume:
  ~12k/hour with ~3% error severity. Anything > 10% error in the recent window is fresh
  degradation."_
- key `noise:logs:rabbitmq-deploy-window` — _"Log message `connection refused: rabbitmq:5672`
  is recurring noise during deploy windows (Mon/Wed 14:00 UTC) — auto-recovers within 5 min."_
- key `pattern:logs:alert-47` — _"Logs alert `db-connection-pool-saturated` (id 47) auto-mutes
  02:00–04:00 UTC for nightly batch — firing outside that window is real."_
- key `addressed:logs:cdp-worker-2026-04-30` — _"Service `cdp-worker` migrated to a new
  runtime on 2026-04-30 — log volume baseline shifted from 8k/hour to 14k/hour, treat new
  baseline as normal."_

By run #5 you'll know per-service volume and severity baselines, which alerts are
intentional outliers, and only surface fresh shifts.

### Decide

For each candidate finding:

- **Emit** via `signals-scout-emit-signal` if it clears the confidence bar.
  Strong scout findings: weight ≥ 0.7, confidence ≥ 0.85, with concrete service /
  message / time-range evidence.
- **Remember** if below the bar but worth carrying forward.
- **Skip** with a one-line note if a scratchpad entry with a `noise:` or `addressed:`
  key prefix already covers it.

If a prior run already covered the topic, default to skip + scratchpad refresh rather
than re-emit. Same fact twice in the inbox degrades signal-to-noise more than missing
one finding for one tick.

### Close out

**Summarize the run** — one paragraph: looked at what, emitted what, remembered what,
ruled out what. The harness writes this to the run row as searchable prose; future runs
read it via `signals-scout-runs-list`. Do **not** write a separate "run metadata"
scratchpad entry — the run summary already serves that role.

## Disqualifiers (skip these)

- **Routine debug logs from internal services** — `severity = debug` records from
  sandbox / internal tooling. Filter before counting.
- **Dev / local / test environment logs** — `service` or attribute values matching
  dev-style patterns (`*-dev`, `*-local`, `*-test`). Filter on the team's expected
  service allowlist.
- **One-off deploy log floods** — temporary spike during a deploy that subsides within
  30–60 minutes. Memory should record the team's typical deploy windows.
- **Logs alerts in muted / snoozed state** — explicit team decision; don't override.
- **Log error already covered by error tracking** — if a log record correlates 1:1
  with an `$exception` issue already surfaced, that issue's finding (or a scratchpad
  entry with `dedupe:` key prefix) governs. Don't double-emit.

When in doubt, write a memory entry instead of emitting.

## MCP tools

Direct calls (read-only):

- `logs-count` — start here. Total volume over a window.
- `logs-count-ranges` — compare windows (today vs 7d-prior, this hour vs same hour
  yesterday); supports breakdowns.
- `logs-sparkline-query` — sparkline shape; useful for spotting a sharp burst vs a
  sustained shift.
- `query-logs` — drill into individual records. Filter by severity, service, message
  text, attribute values, time range.
- `logs-attributes-list` / `logs-attribute-values-list` — discover the team's log shape.
- `logs-alerts-list` / `logs-alerts-retrieve` — configured alerts and current state.
- `inbox-reports-list` — verify a finding isn't already in the inbox.
- `query-error-tracking-issues-list` — cross-check whether a log error already has an issue;
  error tracking owns those findings.

Harness-level:

- `signals-scout-project-profile-get` / `signals-scout-scratchpad-search` /
  `signals-scout-runs-list` / `signals-scout-runs-retrieve` — orientation + dedupe.
- `signals-scout-emit-signal` / `signals-scout-scratchpad-remember` — emit / remember.

## When to stop

- Volume + severity at baseline, no fresh patterns → close out empty.
- A candidate matches a scratchpad entry with `noise:` / `addressed:` / `dedupe:` key
  prefix → skip with a one-line note.
- You've validated some hypotheses and emitted what's solid → close out.

"Looked but found nothing meaningful" is a real outcome.