Investigate An Unfamiliar System

---
name: investigate-an-unfamiliar-system
description: Use when entering a system you did not build. The reading order that gets you oriented without breaking anything, and the false negatives that will fool you on the way.
---

# Investigate an unfamiliar system

You have access to something you have never seen and someone is waiting for an
answer. Two ways this goes wrong:

- You change something to find out what it does. Now you are the incident.
- You conclude something is absent when your instrument simply could not see it.

The second is far more common, far quieter, and produces confident wrong
statements that get written down. Most of this document is about that.

## The one rule for the first hour

**Read-only.** No restarts "to see what happens", no test writes, no removing
something "unused", no restarting a service to check that it comes back. You do
not yet know what depends on it — that is the thing you are here to find out.

Capture as you go. Everything you print in this hour is the starting-state
photograph that any later change is going to need anyway.

## The reading order

Four questions, in this order. Each one is uninteresting until the previous one
is answered.

### 1. What runs, and where

Processes, services, containers, scheduled jobs, listening ports. Not what is
installed — what is **running right now**, and on which machine.

Then, for anything that looks important: how does it get started? Something that
starts on boot, something restarted by a supervisor, and something a person
started by hand in a session six months ago are three very different objects that
look identical in a process list.

### 2. What depends on what

Follow the traffic, not the directory layout:

- What listens, and what connects to it
- What resolves to what — from **inside** the system, which is often not what
  resolves from outside
- Where the data lives: local disk, shared storage, a database somewhere else
- What is in front: anything terminating connections, balancing, caching or
  rewriting before the application sees a request

The useful question at this stage is not "what is this?" but **"what breaks if
this stops?"**

### 3. Who else writes here

🔴 **The assumption that you and the people you can see are the only writers is
the one that hurts.** Automation, deployment pipelines, configuration management,
scheduled jobs, another team, and the system's own supervisor all make changes
that look, in the logs, exactly like a person made them.

Concretely: what were the last modifications, when, and by whom? If a file you
are about to edit is regenerated by something else every ten minutes, your edit
is not a change — it is a delay.

### 4. What is documented, and what is not

Now, and only now, read the documentation — with what you have observed in hand.
Reading it first hands you a frame, and you will spend the rest of the session
confirming it.

For each document ask **when was this last confirmed true**, not when it was
written. Anything describing a component you did not find running is either
wrong, or you missed the component. Both are worth resolving; neither is worth
guessing about.

Whatever is running and appears in no document goes on the list of open
questions, not in the mental bucket of "probably unimportant" — see
`map-an-undocumented-system`.

## The false negatives

Every one of these produces the same output: **nothing found**. And "nothing
found" reads like an answer.

### 🔴 Your instrument may not exist

```
<list-something> | <search-tool> <pattern> || echo "none"
```

If `<search-tool>` is not installed, the pipeline fails, the fallback runs, and
you get `none` — from a missing binary, not from an absent match. The output is
byte-identical to a genuine negative result.

This is the most avoidable wrong conclusion in the whole document, and it is
committed constantly.

**Before you trust any negative:**

1. Confirm the tool exists and is what you think it is.
2. Run the same query for something you **know** is there. If that also comes
   back empty, your instrument is broken and every negative you have collected
   so far is worthless.
3. Separate "the command failed" from "the command found nothing". They are
   different exit conditions and they deserve different reactions.

⚠️ **A negative result is only evidence if the search could have returned a
positive.** Prove it could, once, with a control.

### 🔴 Looking in the wrong registry

Asking the host's package inventory whether a component is installed, when it
runs inside a container, returns a clean, confident "not installed". So does
asking about a service managed by a different supervisor than the one you
queried, or a runtime installed under a user account rather than system-wide.

The tool answered correctly. It answered a different question from the one you
asked.

**Whenever the answer is "it's not there" and you expected it to be there, doubt
the registry before you doubt reality.** Go back to question 1: it either appears
in the list of what is *running* or it does not, and that list does not care how
it was installed.

> An afternoon was spent concluding a component was not installed on a machine.
> It was running the entire time, inside a container, invisible to the host's
> package database. The control that would have caught it in under a minute:
> query the same registry for something known to be running there. It would have
> come back empty too.

### ⚠️ The command you typed is not the command that ran

An interactive shell can carry aliases, functions and path overrides that
silently replace common commands with different implementations — usually
installed deliberately, by someone who liked them better. The name is the same.
The options, the output shape and the behaviour are not.

The visible symptom is a command that "does not work properly" or output in an
unexpected shape. The invisible symptom is a result that looks fine and is
answering a slightly different question.

⚠️ **Ask the shell what a name resolves to before building a conclusion on its
output**, especially on a machine that is not yours.

### ⚠️ Line-oriented searching cannot see across lines

A search tool that examines one line at a time will not find a pattern split
across two, however correct the pattern is.

> A style rule was known to exist. Three searches said it did not. It had been
> reformed onto several lines by a tool, so the sequence being searched for never
> appeared on any single line. The rule was there the whole time; the searches
> were correct and the conclusion was wrong.

Anything structured — nested configuration, serialised data, style rules,
multi-line declarations — needs a search that spans lines, or a parser. Do not
conclude absence from a line-based search over a block-structured file.

### ⚠️ Configuration on disk is not configuration in use

The file you are reading may have been edited after the process started, replaced
by a mechanism that regenerates it, mounted over, or superseded by an
environment variable or a command-line option.

Ask the process what **it** thinks its configuration is. Where it can dump its
own live view, that is the authority. The file is a hypothesis about it.

### ⚠️ One name in front of several nodes

If a name resolves to a load-balanced front, every query you run may land on a
healthy node and every one of your answers describes that node. Nodes drift —
different versions, different config, one that was missed by the last rollout.

Resolve to each address and ask each one individually, before you say anything
about "the system".

## The control step, generalised

Every one of the traps above has the same antidote, and it is worth stating on
its own because it applies to instruments you have not met yet:

> **Point your diagnostic at something whose answer you already know.**

If the method cannot detect a component you are certain is running, a file you
know exists, a pattern you can see with your own eyes — then it cannot detect
anything, and its silence is not information.

Do this once per instrument, at the start. It costs a minute and it is the
difference between "not found" and "not there". `validate-your-validator` is the same
idea applied to anything that returns pass/fail.

## What to write down

```markdown
## <system> — first reading, <date>

**Running:** <processes/services/containers, and on which host>
**Started by:** <boot / supervisor / scheduled / by hand, unknown>
**Depends on:** <what it calls, what calls it, where data lives>
**In front of it:** <proxy, balancer, cache — or "nothing found, see control">
**Other writers:** <automation, pipelines, scheduled jobs, other teams, unknown>
**Documented:** <what exists, last confirmed true when>
**Instruments used and their controls:** <tool → control that came back positive>
**Open questions:** <what I could not determine, and who could answer it>
```

The last two lines are what makes this reading trustworthy to the next person.
Without the controls, every negative in the document is an assertion.

## Traps

🔴 **Do not restart anything to learn what it does.** That is not exploration, it
is an experiment on a production system with no rollback and no notice.

🔴 **"I searched and found nothing" is not a finding until the search has been
shown to work.**

⚠️ **Do not tidy while reading.** A file that looks like leftover junk is a file
whose purpose you have not discovered yet. Write it down as a question instead.

⚠️ **Read the documentation last.** Read it first and you will confirm it rather
than the system.

⚠️ **Absence of alerts is not evidence of health.** A system nobody is monitoring
is quiet for the same reason a healthy one is.

---

Once you understand it well enough to change it, `write-the-rollback-plan-first` and
`deploy-to-production-safely` take over. Whatever you could not determine goes in
the gap list — `map-an-undocumented-system`.