Network Resilience

---
name: network-resilience
description: Use when designing, building, or reviewing any feature that makes a network call for users on intermittent, slow, expensive, or metered connections. Covers offline queueing, idempotency, retry and backoff, sync conflict resolution, timeout sizing, data-saver handling, and graceful degradation on 2G/3G and congested LTE. Applies to mobile and web. Invoke on specs, PRs, API designs, and any code that fetches, posts, syncs, or retries.
license: MIT
---

# Network resilience

## The constraint

The connection is not slow. It is intermittent, asymmetric, and dishonest — it
disappears mid-request, reports itself as live while carrying no traffic to your
origin, and costs the user money per megabyte from a prepaid bundle they top up
in fifty-cent increments. Latency to your origin is 300–800ms before congestion.
The device changes network three times on a matatu ride to work.

Design for a connection that fails *ambiguously*. Total failure is easy: you
show an error. The expensive case is the request that neither succeeds nor
fails, and the user who taps again.

## Hard rules

| ID | Rule | Severity |
|---|---|---|
| NET-001 | Render from local state first; never block first paint on a network call. | critical |
| NET-002 | Queue every state-changing request durably before attempting it, or refuse it — decide per intent, and enforce the decision in one place. | critical |
| NET-003 | Every state-changing request carries a client-generated idempotency key. | critical |
| NET-004 | Pending, queued, and failed states are visible and honest in the UI. | critical |
| NET-005 | Set explicit request timeouts sized for the target network, not the default. | warning |
| NET-006 | Retry with capped exponential backoff and jitter; never tight-loop. | warning |
| NET-007 | Treat the connectivity flag as a hint, not a fact. | warning |
| NET-008 | Honour metered-connection and data-saver signals. | warning |
| NET-009 | Declare a conflict-resolution policy for every entity that syncs. | warning |
| NET-010 | Batch and compress; minimise round trips over payload elegance. | advisory |
| NET-011 | Size a partner integration for their slowest component, not for their API's stated limits. | critical |
| NET-012 | Never let a third party's connectivity probe decide whether your app may call your API. | critical |
| NET-013 | Key every persisted cache to a confirmed account identity, and fence it when the session ends. | critical |
| NET-015 | Distinguish "no data yet" from "stale data", and label cached data with its age. | warning |

Full detection criteria and remedies in [`rules.yml`](rules.yml).

## Judgment

**Offline-first is not free, and it is not always right.** A durable queue costs
storage, a sync engine, and install size — and install size costs you installs
(see `payload-budgets`). The test is whether the user's intent survives the app
being killed. A "like" does not need to; a loan repayment does. Queue the
actions where losing the intent costs the user money, time, or a trip, and let
the rest fail loudly and cheaply — from one place, not eight (NET-002).

**Caching trades staleness for availability, and the exchange rate depends on
the data.** A film recommendation may be a week stale with no harm. A wallet
balance may not be stale at all — but it also must not vanish when the network
does. The resolution is not to pick one: show the cached value, label it with
the time it was fetched, and never let a cached balance authorise a transaction.
NET-001 and NET-004 are the same rule seen from two directions — always render
something, always be honest about what it is.

**Optimistic UI is a claim about the server, so scope the claim.** Optimism is
correct for anything you can reverse without the user noticing, and dishonest
for anything that moves money or grants access. When you cannot be optimistic,
be *fast about being pending*: acknowledge in under 100ms that the intent was
captured. Users tolerate waiting for confirmation. They do not tolerate not
knowing whether they were heard.

**Retry policy is a shared resource.** Your backoff is not only about your
server. Thousands of clients on the same congested cell, all retrying on the
same connectivity-restored event, reproduce the congestion that caused the
failure. Jitter is not politeness; it is the thing that makes recovery possible.

**Do not send the user's money to the background.** Every byte of prefetch,
telemetry, and background sync is drawn from a prepaid bundle. Treat background
data as a budget you are spending on the user's behalf, and be able to state the
per-session number (see `integration-cost-modeling`).

**A cache is an account's data, not the app's.** The moment you persist
anything, the phone holds one user's data across another user's session, and
these markets are where a phone is most likely to be shared, lent, or resold.
The boundary that matters is not logout — it is the confirmed account id. Write
nothing to disk before the server has told you whose data it is, drop the cache
the instant a session ends, and make the sensitive categories unpersistable by
construction rather than by remembering (IDN-011). The failure is silent, it
never appears in single-account testing, and its blast radius is someone else's
earnings.

**"Nothing cached" and "nothing exists" are different sentences.** Collapsing
the two tells a user offline that their work is gone. Decide the ordered states
once — loading, offline-without-data, failed-without-data, empty, content — and
derive them from whether data is present, not from the connectivity flag. Be
honest in the other direction too: cached content without a fetched-at label is
a lie of omission the first time it is a day old.

## Worked patterns

### An outbox that survives process death

Satisfies NET-002, NET-003, NET-006. The key is generated at *intent* time and
persisted with the request, so every retry — including one after a cold start
three hours later — carries the same key.

```ts
// lib/outbox.ts
type Intent = {
  key: string;          // idempotency key, generated once, at intent time
  url: string;
  body: unknown;
  attempts: number;
  nextAttemptAt: number;
};

export async function enqueue(url: string, body: unknown): Promise<string> {
  const intent: Intent = {
    key: crypto.randomUUID(),
    url,
    body,
    attempts: 0,
    nextAttemptAt: Date.now(),
  };
  await idb.put("outbox", intent);       // durable before any network attempt
  void drain();                           // fire and forget; drain is idempotent
  return intent.key;                      // caller renders "queued" against this
}

export async function drain(): Promise<void> {
  for (const intent of await idb.getAll("outbox")) {
    if (intent.nextAttemptAt > Date.now()) continue;

    try {
      const res = await fetch(intent.url, {
        method: "POST",
        headers: {
          "content-type": "application/json",
          "idempotency-key": intent.key,   // same key on every attempt
        },
        body: JSON.stringify(intent.body),
        signal: AbortSignal.timeout(30_000), // NET-005: patient, writes must land
      });

      if (res.ok) {
        await idb.delete("outbox", intent.key);
      } else if (res.status >= 400 && res.status < 500) {
        await deadLetter(intent, await res.text()); // will never succeed; stop
      } else {
        await backoff(intent);
      }
    } catch {
      await backoff(intent);               // timeout or transport failure
    }
  }
}

async function backoff(intent: Intent): Promise<void> {
  if (intent.attempts >= 8) return deadLetter(intent, "retry cap reached");
  const ceiling = Math.min(2 ** intent.attempts * 1_000, 5 * 60_000);
  await idb.put("outbox", {
    ...intent,
    attempts: intent.attempts + 1,
    nextAttemptAt: Date.now() + Math.random() * ceiling, // NET-006: full jitter
  });
}
```

Drain on app start, on the connectivity-restored event, and from a periodic
background task. On Android, the equivalent is a `CoroutineWorker` under
`WorkManager` with `setBackoffCriteria(BackoffPolicy.EXPONENTIAL, ...)` and a
`NetworkType.CONNECTED` constraint — the platform already solved persistence
across process death, so do not rebuild it.

### Proving the network before trusting it

Satisfies NET-007. `navigator.onLine` answers "is there an interface up", which
is not the question.

```ts
export async function originReachable(): Promise<boolean> {
  if (!navigator.onLine) return false;              // fast negative is reliable
  try {
    const res = await fetch("/api/ping", {
      cache: "no-store",
      signal: AbortSignal.timeout(4_000),
    });
    // A captive portal or operator upsell page returns 200 with HTML.
    return res.ok && res.headers.get("content-type")?.includes("json") === true;
  } catch {
    return false;
  }
}
```

The content-type check is the point. A depleted prepaid bundle terminates your
request at an operator portal that answers `200 OK` with a top-up page. Status
codes alone will tell you everything is fine.

### Degrading by measured conditions, not by guess

Satisfies NET-001, NET-008. Serve the cheap thing by default and upgrade only
when the connection has earned it.

```tsx
function useAssetTier(): "minimal" | "standard" | "rich" {
  const c = (navigator as any).connection;
  if (!c) return "standard";
  if (c.saveData) return "minimal";                 // explicit user instruction
  if (["slow-2g", "2g"].includes(c.effectiveType)) return "minimal";
  if (c.effectiveType === "3g") return "standard";
  return "rich";
}
```

Tie the tier to image resolution, prefetch, autoplay, and animation — one signal
driving every expensive decision, rather than each component deciding alone.
`saveData` is a user instruction, not a hint; there is no tier above `minimal`
when it is set.

### A cache boundary that survives a shared phone

Satisfies NET-013, NET-015, IDN-011. The persister is created for a confirmed
account id and closed before the cache is cleared; the provider is remounted so
no observer outlives the session.

```ts
// Persist only what is safe on a device someone else may hold (IDN-011).
const PERSISTABLE = ['jobs', 'earnings', 'messages', 'notifications'];
export const shouldPersist = (key: readonly unknown[]) =>
  PERSISTABLE.includes(String(key[0]));

export const persisterFor = (accountId: string) => {
  const storage = new MMKV({ id: `cache.${accountId}` });
  let closed = false;
  const write = debounce((state: string) => {
    if (closed) return;              // a scheduled write must not outlive the session
    storage.set('cache', state);
  }, 1000);
  return {
    persist: (state: string) => write(state),
    close: () => { closed = true; write.cancel(); },
    remove: () => { closed = true; write.cancel(); storage.clearAll(); },
  };
};

// The session boundary: a new client per session, and a remount so old
// observers cannot serve the previous account.
const { generation, client } = useSessionQueryClient(accountId);
return (
  <QueryClientProvider key={generation} client={client}>
    {children}
  </QueryClientProvider>
);
```

Swapping `client` on an existing provider is not enough: mounted observers keep
the client they subscribed to, and they will happily render the previous
account's data until something forces them to re-render.

## Anti-patterns

**Retrying on the connectivity-restored event with no jitter.** Every client on
the cell fires at the same instant, the cell saturates, all requests time out,
and every client schedules another synchronised retry. You have built a
self-inflicted DDoS that recovers only when users close the app.

**Generating the idempotency key at send time.** The key must be born with the
intent. Generated inside the retry loop, it changes on every attempt, and the
server sees a duplicate as a new legitimate request. The user pays twice and the
logs show two clean successes.

**Reverting optimistic state silently on failure.** The item was in the cart,
then it was not. Nothing was said. The user assumes they misclicked, redoes it,
and it fails again. Every optimistic update needs a failure path that names what
happened.

**A spinner with no timeout.** On an ambiguous connection it spins for ninety
seconds. The user force-quits, losing any in-memory queue, and reopens to find
their action gone. Every indeterminate state needs a deadline and an exit.

**Asking Google whether you can reach your own server.** The connectivity
libraries default their reachability probe to `generate_204`, and Android's own
"validated" verdict is the same fetch. On a link where that probe is slow or
blocked — a carrier that throttles Google, a tether, a portal — the flag says
"unreachable" while your API answers in a second. If a request path checks that
flag before trying, your app is the only one on the phone that stops working.
Probe an endpoint you own, and never let any probe veto a request: the request
is the test.

**Treating a cached balance as authoritative.** Cached values are for display.
The moment one is used to authorise a transaction, staleness becomes an
overdraft. See `money-movement` for the server-authoritative pattern.

**Shipping a sync engine before shipping the feature.** Offline-first is
architecture, and architecture chosen before you know which actions matter
produces a 14MB app that syncs everything and installs on nobody's phone.
Identify the two or three intents that must survive, queue those, and grow.

**Clearing the cache on logout and calling it a boundary.** The write was
already scheduled. It lands a second later, after the clear, and the next person
to open the app sees the last person's balance. Close the writer before you
clear, and test it by logging out and straight back in as someone else.

**Treating an empty cache as an empty account.** "No jobs yet" to a creator who
has eleven jobs and no signal is not a cosmetic bug; it is the app telling them
their work is gone, at the moment they are least able to check.

**Every screen inventing its own offline behaviour.** One queues, one fails
loudly, one fails silently, and eight of them raise the same toast at once on a
cold start. Refusing an offline write is a legitimate answer to NET-002 — for
most intents it is the right one — but it is a decision to make per intent and
implement once, not a gap for each screen to fill.

## Verification

```bash
python3 skills/network-resilience/scripts/audit_handlers.py <path>
```

Flags state-changing handlers with no queue write, missing idempotency headers,
`fetch` calls with no timeout, and branches on `navigator.onLine` with no
confirmation round trip.

Static analysis will not tell you whether it works. Also:

- **Test on a throttled link, not a disabled one.** Airplane mode exercises the
  easy path. Use `tc netem` or Chrome DevTools at 400kbps with 400ms RTT and 2%
  packet loss, and drop the connection *mid-request*, which is the case that
  finds bugs. See `field-testing-and-telemetry`.
- **Kill the process during a queued write.** On Android, `adb shell am kill`.
  The intent must be there on restart.
- **Black-hole the OS connectivity probe and use the app.** On the emulator,
  `adb shell settings put global captive_portal_https_url http://10.255.255.1/`
  (and the `http`/`fallback` keys), cycle Wi-Fi, disable mobile data so the
  unvalidated network is the default. The app must still reach your API. Reset
  the keys afterwards.
- **Instrument the ambiguous case.** Track time-to-confirmation, queue depth,
  dead-letter rate, and duplicate-intent rate segmented by effective connection
  type. A duplicate-intent rate above zero means NET-003 is not holding
  somewhere.

## Sources

- NET-001, NET-003 — field, Juvo Mobile (6 markets, Central & South America,
  2015–2020).
- NET-002 — field, gethomespace (Kenya/US, 2020–2022) and Wowzi (Kenya,
  2026); vendor, Android WorkManager.
- NET-004 — field, IBM Research Africa (Kenya, 2013–2014).
- NET-006 — vendor, AWS Architecture Blog on exponential backoff and jitter.
- NET-007 — field, Smile Identity (pan-African, 2017–2020).
- NET-008 — vendor, Chrome `Save-Data` and Network Information API.
- NET-012 — field, Wowzi (Kenya, 2026).
- NET-013, NET-015 — field, Wowzi (Kenya, 2026).

Full citations in [`docs/SOURCES.md`](../../docs/SOURCES.md).