Fetch Url

---
name: Fetch URL
description: HTTP GET/POST utility with size cap, SSRF guard, retries, and circuit breaker.
requires:
  bins: []
  env: []
---

# Fetch URL

Use this skill to download remote HTTP(S) content — raw API calls, web
pages, JSON endpoints, small files — safely. Private and metadata hosts
are blocked by default to prevent SSRF.

## Use when

- "Fetch this URL"
- "What does this API return?"
- Pipeline: fetch → `pdf-extract` → `summarize_text`
- Fetch a webhook payload or a small data file
- Check a public HTTP endpoint (status + headers)

## Do not use when

- Downloading binaries/archives > 50 MB (hard cap; the LLM context can't
  hold them anyway)
- Long-poll / streaming / SSE / WebSocket — this is one-shot request/response
- Authenticated APIs with complex OAuth flows — use a purpose-built
  extension (github, whisper, summarize) that owns the auth
- Calling internal services — set `allow_private: true` only on
  explicitly trusted operator calls

## Tools

### `status`
No arguments. Returns limits, policy, default timeout, user-agent.

### `fetch_url`
- `url` (string, required) — `http://` or `https://` only
- `method` (string, optional) — GET/POST/PUT/DELETE/HEAD/PATCH/OPTIONS (default GET)
- `headers` (object, optional) — `{"content-type": "application/json", ...}`
- `body` (string, optional) — request body as raw string; caller sets Content-Type
- `max_bytes` (integer, optional) — response cap. Default 5 MB, hard max 50 MB
- `timeout_secs` (integer, optional) — per-request timeout. Default 15s, max 120s
- `allow_private` (boolean, optional) — override SSRF guard. Default false

Returns `{status, final_url, headers, content_type, truncated, bytes_read, body_text?, body_base64?}`.

`body_text` is set for text-like content-types (text/*, json, xml, yaml,
javascript) if UTF-8 decodable. Otherwise `body_base64` is set and
`body_text` is null.

## Execution guidance

- **Start with a small `max_bytes`** (e.g. 100 000) when probing an unknown
  endpoint — you can always retry with a larger cap.
- Default timeout (15s) is fine for APIs; bump to 60–120s only for large
  files.
- If `status` is 4xx (-32002 error), the body preview is in the error
  message — read it before retrying.
- If `status` is 5xx (-32003), retries already exhausted inside the
  extension; do **not** retry from the LLM side — back off instead.
- `blocked_host` error (-32020) → the URL points to a private IP /
  loopback / metadata endpoint. Do **not** pass `allow_private=true`
  unless the operator explicitly asked to hit an internal service.
- `truncated: true` means the body was cut at `max_bytes`. Warn the user
  or re-fetch with a bigger cap if needed.
- For multi-step fetch-and-process, wrap the chain in TaskFlow so a
  restart doesn't re-fetch.

## Common pipelines

### URL → summary

```
1. fetch_url { url, max_bytes: 50000 }
2. summarize_text { text: body_text, length: "medium" }
```

### URL → PDF → summary

```
1. fetch_url { url: "https://.../doc.pdf", max_bytes: 5000000 }
2. (operator saves body_base64 to /tmp/doc.pdf)
3. pdf-extract.extract_text { path: "/tmp/doc.pdf", max_chars: 50000 }
4. summarize_text { text, length: "long" }
```

(Auto-saving to disk from inside the LLM is not yet supported — a future
extension `fetch_url_save` could do that safely.)