Logql Observability

---
name: logql-observability
description: "Queries workload logs with LogQL on Control Plane. Use to troubleshoot a workload from its logs, or for log search, access or egress logs, cron run logs, missing or truncated logs, retention, or logs in Grafana."
---

# LogQL & Log Observability

Control Plane stores workload stdout/stderr in Loki and queries it with LogQL. The org is the Loki tenant — it comes from the endpoint path, so `org` is never a query label and queries cannot cross orgs. Reading logs requires the org-level `readLogs` permission, and the in-pod `CPLN_TOKEN` cannot authenticate to the logs endpoint — use a user or service-account token (see the `workload` skill). The recurring agent failure is passing a raw `query` to the MCP tool alongside structured params: a raw query replaces them entirely (the tool rejects the combination), so a raw query must embed every label itself.

## Two ways to query

- **MCP (primary for agents):** `mcp__cpln__get_workload_logs` — structured params `gvc` (required), `workload`, `container`, `location`, `filter` (literal substring, LogQL `|=`, not regex); window `since` (default `1h`) or `from`/`to` (ISO 8601, `from` inclusive, `to` exclusive); `limit` (default 30, max 999, single request — `truncated: true` means narrow the window or filter harder); `order` (`oldest_first` default, or `newest_first`). For regex, parsers, or the `replica`/`stream`/`version` labels, pass a raw `query` (max 500 chars) instead of the structured selectors.
- **CLI:** `cpln logs '<LOGQL>'` — interactive debugging (live `--tail`), scripts, CI.

```bash
# Defaults: --since 1h, --limit 30, --direction forward
cpln logs '{gvc="GVC", workload="WORKLOAD"}' --org ORG
cpln logs '{gvc="GVC", workload="WORKLOAD"} |= "error"' --limit 100
cpln logs '{gvc="GVC", workload="WORKLOAD", container="main"}' --since 7d
cpln logs '{gvc="GVC", workload="WORKLOAD"}' --tail              # live follow; the server ends a tail session after 30m
cpln logs '{gvc="GVC", workload="WORKLOAD"}' \
  --from 2026-06-01T00:00:00Z --to 2026-06-02T00:00:00Z          # ISO 8601 or relative (7d, now-1M); from inclusive, to exclusive
cpln logs '{gvc="GVC", workload="WORKLOAD"} |= "error"' --since 24h --limit 0   # 0 = unlimited, auto-paginates
cpln logs '{gvc="GVC", workload="WORKLOAD"}' -o jsonl            # one JSON object per line; -o raw = bare lines
```

- `--since` takes relative durations (`ms s m h d w mo y`, compound like `1h30m`). `--from`/`--to` (CLI) take an ISO 8601 timestamp **or** a relative duration meaning that long ago — `--from 7d`, `--from now-1M`, `--to now-30m` (the CLI accepts `M` for months; the `mcp__cpln__get_workload_logs` tool takes ISO 8601 only).
- `cpln workload eventlog WORKLOAD` (alias `cpln workload log`) is resource event history, not container output — for application logs always use `cpln logs`.

## Labels

| Label | Value |
|:---|:---|
| `gvc` | GVC name |
| `workload` | Workload name |
| `container` | Container name, or a built-in stream below |
| `location` | Deployment location, e.g. `aws-us-east-1` |
| `provider` | Cloud provider |
| `replica` | Replica (pod) name — unique per cron execution |
| `stream` | `stdout` or `stderr` |
| `version` | Workload deployment version that wrote the line |

At least one non-empty matcher is required; regex matchers work — `{gvc=~".+"}` spans every GVC in the org.

## Filters and LogQL features

| Operator | Meaning | Example |
|:---|:---|:---|
| `\|= "text"` | contains | `\|= "error"` |
| `!= "text"` | does not contain | `!= "health"` |
| `\|~ "regex"` | matches regex | `\|~ "timeout\|crash"` |
| `!~ "regex"` | does not match | `!~ "debug\|trace"` |

Loki is current (3.x), so full LogQL works: parsers (`| json`, `| logfmt`, `| pattern`, `| regexp`), post-parse label filters (`| latency > 100`), `line_format`, and metric queries (`count_over_time`, `rate`, `sum ... by`). The CLI and MCP tool print log lines only — run metric queries in Grafana: the `Explore on Grafana` link on the console Logs page opens it with the query prefilled (the org `grafanaAdmin` permission grants the Grafana Admin role; everyone else is Viewer).

```logql
{gvc="GVC", workload="WORKLOAD"} |= "error" != "health"               # errors minus noise
{gvc="GVC", workload="WORKLOAD"} |~ "panic|fatal|exception"           # crashes and stack traces
{gvc="GVC", workload="WORKLOAD", container="_accesslog"} |= "\" 50"   # HTTP 5xx in access logs
sum(count_over_time({gvc="GVC", container="_accesslog"} |= "\" 50"[1m])) by (workload)   # 5xx rate (Grafana)
```

## Built-in log streams

| Selector | Contents |
|:---|:---|
| `container="_accesslog"` | Inbound requests (Envoy access-log format) on the workload's ports |
| `container="_requestlog"` | The workload's outbound (egress) requests through the sidecar, same format |
| `workload="_loadbalancer"` | Access logs of the GVC's dedicated load balancer |
| `container="_alerts"` | Threat-detection (Falco) alerts, with extra labels `rule`, `priority`, `source` |

Platform health probes and unroutable-request noise are filtered out of `_accesslog` by design, so probe traffic never shows up there. Sidecar and system containers (`istio-init`, `istio-validation`, `cpln-*`, `debugger-*`) are never collected.

## Why logs go missing (pipeline limits)

- **Lines over 16 KiB are cut** at 16 KiB with a `... [truncated N bytes]` suffix; empty lines are dropped.
- **Per-replica rate limit:** each replica+container pair is capped (roughly 10,000 lines/s on managed clusters; effectively unlimited on BYOK). Excess lines are dropped, and when collection resumes one marker line appears: `#### Replica logs were rate-limited: N lines in the last Xs were not collected ####`.
- **Retention:** org spec `observability.logsRetentionDays`, default 30 (0 turns log collection off entirely; the `org-management` skill owns org spec edits). Queries beyond retention return nothing, not an error.
- **Server caps:** one query may span at most 31 days and times out after 2 minutes; tail sessions end after 30 minutes.

## Cron workloads: logs for one execution

`{gvc=, workload=}` on a cron workload interleaves every past run. Each execution runs in its own replica, so scope to one run with the `replica` label plus the execution's time window.

**1. List executions.** `mcp__cpln__list_deployments` **with** `location` returns the full deployment JSON; `status.jobExecutions[]` holds the per-execution metadata (without `location` the tool returns only a readiness summary). CLI:

```bash
cpln workload get-deployments WORKLOAD --gvc GVC -o json | jq '
  [(.items // .)[] as $d | $d.status.jobExecutions[]? | . + {location: $d.name}]
  | sort_by(.startTime)[] | {location, name, status, startTime, completionTime, replica}'
```

Schema facts (nodelibs `cronjob.ts`) that bite scripts:

- `status` is one of `successful | failed | active | pending | invalid | removed` (default `pending`). Running means `status == "active"` AND no `completionTime`; a missing `completionTime` alone proves nothing (the schema warns it is not an indication of success).
- `completionTime: null` is stripped server-side and `startTime` may be absent for never-started runs — each field is present-with-value or absent, so guard flags with `${VAR:+--from "$VAR"}` and jq `// empty`.
- `replica` is optional: when missing, the run never got a pod and there are **no logs** — diagnose with the execution's `containers` map and `message` field (aggregated pod events) and `mcp__cpln__get_workload_events`.

**2. Query that replica, time-bounded.** MCP: raw `query` embedding ALL labels (structured params must be omitted) plus `from`/`to`. CLI:

```bash
cpln logs '{gvc="GVC", workload="WORKLOAD", location="LOCATION", replica="REPLICA-ID"}' \
  --org ORG --from START_TIME --to COMPLETION_TIME
```

`jobExecutions` timestamps are ISO 8601 and pass straight through. Pad the window a minute or two on each side, and widen it before concluding logs do not exist. For a live run (`active`, no `completionTime`), drop `--to` and add `--tail`.

## CLI fallback

CI/CD and headless use: set `CPLN_TOKEN` and run `cpln logs` directly (no profile needed); the principal must hold org `readLogs`.

## Troubleshooting

| Symptom | Cause and fix |
|:---|:---|
| 403 `requires permission "readLogs" in org` | Grant `readLogs` on the org via policy (it implies `view`) |
| `Invalid --from format: ...` | `--from`/`--to` accept an ISO 8601 timestamp or a relative duration (`7d`, `now-1M`); correct the value |
| Empty result for known activity | Window outside retention (default 30d), span over 31 days, or filters too narrow; app may not write to stdout/stderr |
| `#### Replica logs were rate-limited ... ####` marker | Per-replica cap was hit; lines in that interval are gone — reduce log volume |
| Line ends with `... [truncated N bytes]` | 16 KiB per-line cap — emit smaller lines |
| Health checks absent from `_accesslog` | Filtered by design; probe failures surface in `mcp__cpln__get_workload_events` |
| MCP rejects raw `query` combined with `workload` etc. | A raw query replaces the structured params — embed all labels in the query itself |
| `cpln workload log` shows no app output | That is the eventlog alias; use `cpln logs` |