Searchmcp Research

---
name: searchmcp-research
description: Getting the most out of SearchMCP research — one web-research call instead of a chain of webfetches. How to prepare the query, choose sources/engines/max_chars, use cache-first repeats, and read the digest, statuses and meta. Citation discipline (URLs, not synthesis), fact verification (2+ sources), recipes per task type, a self-check before answering, and large volumes (30–60 sources, SERP pagination, human pacing). Use for any search, fact gathering, source comparison, current news, claim checking or large collection.
license: MIT
compatibility: opencode, Claude Code, Cursor, OpenAI Codex, VS Code (Copilot), Gemini CLI, Antigravity CLI, Crush, Pi, Kimi Code CLI, Hermes Agent
metadata:
  category: tool-usage
  complexity: intermediate
  author: SearchMCP
  version: "1.4.0"
  tool_macro: searchmcp
---



# SearchMCP Research — get the most out of one call

SearchMCP is a local MCP server: a single `research` call queries several search
engines through an anti-detect browser (Camoufox), opens up to N sources in
parallel and returns a **Markdown digest plus JSON with per-source statuses**.
No paid APIs, everything local.

**The main principle:** research = 1 `research` call (discovery + extraction) →
targeted `webfetch` calls for deep reading of 1–3 pages. Do not fire dozens of
blind `webfetch` calls.

## Universal access to the research tool (never hardcode)

Before calling, resolve the real name: find the tool by description ("research",
search) in the available-tools list. Prefixes: `tools.searchmcp.research`
(opencode/opencode2), `mcp__searchmcp__research`
(Claude Code/Cursor/Codex/Copilot/Kimi/Hermes), `mcp_searchmcp_research`
(Gemini CLI/AGY/Crush, FQN form). On an unknown client, take the real name from the
environment — do not assume it.

## When to use

- Searching for or gathering facts on a topic where you need a view across 3+ sources.
- Comparing opinions or data, fact-checking, market overviews.
- Current news or volatile data.
- When it is unclear where the answer lives — the tool finds the best pages itself.
- Refining the same topic again (cache → instant, no browser).

## When NOT to use

- You need the full text of one **known** URL — call `webfetch` directly (research truncates text).
- Social networks and video (youtube, twitter/x, facebook, instagram, tiktok) — denylisted, come back `skipped`.
- PDF or binary files — `unsupported_content_type` (PDF parsing is out of v1).
- Internal or local addresses — `blocked_ssrf` (SSRF protection).

## The API in one call

```python
mcp__searchmcp__research(
    query: str,                     # 1–500 characters, required
    sources: int | None = None,     # 1–40, default 20 (clamped)
    engines: list[str] | None = None,  # duckduckgo, bing, brave, google; None = all from config
    max_chars_per_source: int | None = None,  # 0–20000, default 4000 (0 = no truncation)
    fresh: bool = False,            # True = bypass the read cache (full pass)
) -> dict
```

The response: `query`, `requested_sources`, `total_sources`, `digest_markdown`,
`sources[]`, `meta`.

## How to read the response

### `digest_markdown` — the report, already assembled
- **Overview** — headings plus quotes from the top 3 sources (a quick look).
- **Key sources (N)** — one block per `ok` source: URL, author/date, quote, the
  markers `_from cache_` / `_⚠ stale copy_`, and which engine produced the link.
- **Problem sources** — URL plus the reason.
- **Metadata** — engines (status: count), timing, cache counters.

### `sources[]` — the structure for targeted work
Useful fields: `url`, `title`, `snippet`, `excerpt`, `outline[]`, `key_points[]`
(up to 3 paragraphs, **the most relevant to the query** — this is what you quote),
`author`, `published_date`, `status`, `engine`, `is_cached`, `cache_age_s`, `error`.

**Quoting rule:** `key_points[0]` → `excerpt` → `snippet`. Never invent a quote from
a title.

### Citing — URLs, not synthesis (critical)

The digest and any summary you write are a **bridge to the data, not a source of
data**. Only concrete URLs from `sources[]` may be cited:

- ✅ "According to docs.python.org [1], asyncio.Semaphore…"
- ✅ "The X report [2] and the Y review [3] state…"
- ❌ "According to the research, Semaphore…" (no link — a hallucination-shaped citation)
- ❌ "As the digest says…" (the digest is your tool, not a source)

The shape of the final answer after `research`:

1. **TL;DR** — 2–3 sentences on what was found.
2. **Facts** — each with a `[n]` link to its source (numbering = order in `sources[]`).
3. **Disagreements / open questions** — when sources contradict each other.
4. **Sources** — the full `[n] URL — title` list, so it can be checked.

### `meta` — call diagnostics
- `engine_results`: `[{name, status, results_count}]`, `status ∈ ok|blocked|error|cooling_down`.
- `cache`: `page_hits`, `serp_hits`, `page_fetched`, `served_stale`, `denylist_skipped`, `cooldown_skipped`.
- `truncated` (true = there were more sources than were processed), `duration_ms`, `sources_ok`.

## Statuses — do not panic, read them correctly

### Source statuses (in `sources[]`)

| Status | Meaning | Action |
|---|---|---|
| `ok` | Content extracted | Use it |
| `stale` | A stale copy served from cache (the page failed) | Use it, but check freshness; for news use `fresh=true` |
| `timeout` / `failed` | The page did not load | NOT "no data" — just retry later |
| `blocked_ssrf` | Private IP or domain | Will not retry; this is protection |
| `unsupported_content_type` | PDF or binary | Look for an HTML version |
| `skipped` | Denylist (social, video) | Noise for research, ignore |

### Engine statuses (`meta.engine_results` / "Engines" in the digest)

| Status | Meaning | Action |
|---|---|---|
| `ok` | SERP retrieved | — |
| `blocked` | Captcha or block on the SERP | NOT "no results"! The engine cools down for 10 min while the others worked |
| `error` | Adapter or parsing failure | The engine is dropped, its budget is redistributed |
| `cooling_down` | Resting after a recent block | Wait, or narrow `engines` |

## Fact verification — the confirmation threshold

Web research returns **raw material, not truth**. How to present it:

| Situation | Presentation |
|---|---|
| The fact is confirmed by **2+ independent** sources | "According to [1] and [2], …" — state it confidently |
| **A single source** | "According to [3] (the only source), …" — with a caveat, not as fact |
| Sources **contradict** each other | A separate "Disagreements" block: [1] says X, [2] says Y; do not average |
| The topic was found by **several engines** (`meta.engine_results`: 3–4 × `ok`) | Higher confidence in the coverage — not a random hit |
| The topic was found by **one engine** (`ok`=1) | Be careful: possibly a narrow or exotic query — verify differently |

**Anti-duplicate:** if two sources are reprints of one text (an aggregator copying a
news outlet), they are NOT two independent sources; find the original.

## Ranking sources — what to trust

All else being equal, rank top-down and cut reprints and SEO farms:

| Source type | Weight | How to present |
|---|---|---|
| Primary: official docs, `docs.*`, `*.gov`/`*.edu`, repositories, primary data (APIs, statistics) | Highest | The citation that carries the claim |
| Authoritative media, reviews, expert blogs | Medium | Confirmation and context |
| Aggregators, forums (Reddit and similar), copy-paste farms, as-is articles | Low | A lead or a direction, not a basis; better to cite the original inside |
| Social networks and video (youtube/x/facebook etc.) | Not a source | The server denylists them — do not try to pull them |

Practical rule: before answering, filter out noisy URLs by pattern (aggregators,
spam domains, forums) and keep the 3–5 best per the table above — the answer becomes
shorter and more trustworthy.

## Sizing the call to the task (token budget)

20 sources × 4000 characters ≈ 80K characters ≈ **20–25K tokens** per call. Set the
parameters deliberately:

| Task | sources | max_chars_per_source | engines |
|---|---|---|---|
| Quick answer / fact-check | 4–6 | 1200–2000 | `["duckduckgo","bing"]` |
| Standard research (default) | 8–12 | 4000 | all |
| Deep dive into a topic | 15–20 | 6000 | all |
| Comparative overview / "what do people say" | 10–15 | 1500–2500 | all |
| Fresh news | 6–10 | 3000 | `["google","bing"]` + `fresh=true` |
| You need full texts (few URLs) | 3–5 | 20000 | all |

## Strategies for maximum effectiveness

### 1. Decompose a big question — 2–4 parallel calls
One giant query is not depth. For "compare the market for X", run 3 calls in
parallel: subtopic A, subtopic B, subtopic C. Each call with its own `sources`
(5–10) finishes faster and more completely than a single call with 40.

### 2. The cache — a cascade of cheap refinements
- The SERP is cached for 24h, pages for 7 days; a repeated query is **instant, no browser**.
- **Diversity**: new URLs load first and the cache fills the remainder, so each repeat
  of the same query widens coverage (just adding `sources` is the better lever).
- `fresh=true` — **only** when freshness is critical (news, prices, volatile data).
  Otherwise use the cache.
- A plan: `research(Q, sources=10)` → look → `research(Q, sources=20)` (picks up new
  sources from cache almost for free).

### 3. Write the query for the engines and the region
- The more specific the query (2–4 meaningful terms, "exact phrases"), the better the
  extractor isolates `key_points`.
- Operators work per engine: `site:domain.com`, quotes, negative terms; `site:` is most
  reliable on Google and Bing.
- One call = one topic. A multilingual topic → 2 calls, one per language (engines are
  regionally different).
- When an engine is blocked, do NOT immediately re-query the same one — the cooldown is
  10 min; narrow with `engines=["duckduckgo","brave"]` instead.

### 4. Research for discovery → webfetch for exact text
The digest is truncated (`max_chars`). If you need the full text of 1–3 pages: take
URLs from `sources[]` (use `key_points`/`outline` to choose) → `webfetch` those pages.
You pay tokens only for what you need.

### 5. Google/Bing for quality, DDG/Brave for reliability
- Fast and stable: `duckduckgo, bing`.
- Maximum relevance on the English-speaking web: add `google`.
- Do not pass unknown engines — the whole call fails with an error.

## Antipatterns (don't)

- ❌ 10 separate webfetches instead of one research.
- ❌ `sources=40, max_chars=20000` "just in case" — a token explosion.
- ❌ `fresh=true` every time — you lose the cache and the speed.
- ❌ Treating `blocked`/`timeout` as proof that no data exists.
- ❌ Asking research for social or video content (denylist).
- ❌ Raising `SEARCHMCP_CONCURRENCY` above 1 for speed — it flakes (empirical, see README).

## Recipes per task type

Do not redesign this every time — start from a ready pipeline.

### R-1: Fact-checking a claim
1. `research(claim, sources=5, max_chars_per_source=2000, engines=["duckduckgo","bing"])`.
2. Read the `key_points` of the top sources → find confirmation or refutation.
3. ≥2 independent → "confirmed [1][2]"; 1 → "according to [1] (the only source)";
   a contradiction → a "Disagreements" block.

### R-2: Comparing opinions / "what do people say"
1. Split the subject into 2–3 positions → **parallel** calls (one per position),
   `sources=10–15, max_chars=1500–2500`.
2. Per position: who says it → 1–2 best sources each (see "Ranking").
3. The answer: position A — [1][2]; position B — [3]; the disagreements between A and B.

### R-3: Fresh news / trends
1. `research(query, sources=6–10, max_chars=3000, engines=["google","bing"], fresh=true)`
   — freshness is critical.
2. Check `published_date` — drop reprints older than 2–3 days and find the original.
3. Answer with dates: "[date] — [the gist] ([source])" plus a block
   "what is certain / what is changing".

### R-4: Deep research on a topic
1. `research(topic, sources=15–20, max_chars=6000)` — the survey pass.
2. `outline[]`/`key_points[]` → choose 2–4 lines to deepen → parallel `research` calls
   on the subtopics (or let `sources` pull more from cache).
3. Full texts of 1–3 key pages → `webfetch` by the URLs from `sources[]`.
4. Output: TL;DR → facts with `[n]` → disagreements → sources (the format above).

### R-5: Large volume (30–60 sources) — 1–2 calls, no blocks
1. Switch on the bulk mode (env, once): `SEARCHMCP_SERP_PAGES=2–3` (SERP pagination,
   ×2–3 candidates), `SEARCHMCP_SOURCES=50` (limit 60), `SEARCHMCP_BATCH=8` (waves),
   `SEARCHMCP_PACING_MS=400` (human pacing — keep the default; it is the antidote to
   captchas).
2. First call: `research(topic, sources=30–40, max_chars=2000)` — discovery plus top
   content in a compact form.
3. Second call (same query, `sources=50–60`): cache top-up — new URLs cost almost
   nothing because the SERP and pages are cached.
4. Token budget: 60 sources × 2000 ≈ 120K characters ≈ 30K tokens per call — fine for
   large collections; pull full texts only where needed (`webfetch`).
5. Anti-block checklist: pacing is on (pacing > 0); when an engine reports `blocked`,
   do not repeat immediately (10 min cooldown), narrow `engines`; on timeouts, repeat the
   call and let the cache catch up; never set `SEARCHMCP_CONCURRENCY>1` (it flakes).

### Advanced env (bulk + anti-block, server side)

| env | Default | Why |
|---|---|---|
| `SEARCHMCP_SERP_PAGES` | 1 | 2–3 = SERP pagination (×2–3 candidates) for large collections |
| `SEARCHMCP_PACING_MS` | 400 | random 200–600 ms pauses between requests (human pacing; 0 = off) |
| `SEARCHMCP_BATCH` | 8 | page-load waves (even load, fewer blocks) |
| `SEARCHMCP_RETRY_BACKOFF_S` | 1.5 | jittered pause before retrying a failed page |
| `SEARCHMCP_SOURCES` | 20 | target number of sources (1–60) |

## Self-check before the final answer

Walk the checklist — if any line is "no", fix the answer:

- [ ] Are all `ok` sources with a "noise" weight filtered out (see Ranking)?
- [ ] Does every fact have ≥2 independent sources, or is it marked "single source"?
- [ ] Are contradictions in their own block rather than averaged away?
- [ ] Is every quote tied to a source URL `[n]`, none taken from the digest or a summary?
- [ ] Were the statuses read — `blocked`/`timeout` not treated as "no data"?

## Troubleshooting

| Symptom | Cause → fix |
|---|---|
| Every engine `blocked` | Captchas from the local IP → wait out the cooldown, try `engines=["duckduckgo"]`, use a proxy (`SEARCHMCP_PROXY_URL`), or `fresh=false` (the cache may still serve a SERP) |
| Empty or thin digest | The query is too narrow or exotic → widen it or switch engines |
| The call is slow | A first pass is a full pass (10–30 s plus page loads); a repeat hits the cache. Check `meta.duration_ms` |
| `cache stats` hints at corruption | Falls back to a cache-less mode automatically; just repeat |
| Unsure about the environment | `python -m searchmcp doctor` |

## CLI equivalents (for a human or a script)

```bash
python -m searchmcp research "python asyncio semaphore" --sources 10
python -m searchmcp research "topic" --engines duckduckgo,bing --max-chars 2000 --format markdown
python -m searchmcp research "topic" --fresh
python -m searchmcp cache stats
python -m searchmcp doctor
```

## Skill activation

The skill is published by the `skills_install.py` farm: the canon lives at
`SearchMCP/skills/searchmcp-research/SKILL.md` and macros generate per-client
derivatives (`~/.config/opencode/skills/searchmcp-research`, `skills/`,
`~/.agents/skills/`, `skills/`). Triggers: "search / find facts / compare
sources / what do people say about X", claim checking, fresh news.

## SIFT source evaluation & hypothesis cycle

1. **SIFT** (Michael Caulfield): **S**top → **I**nvestigate the source → **F**ind better independent coverage → **T**race claims to the primary source. Require 2+ independent sources for key claims.
2. **Research spiral:** RESEARCH → HYPOTHESIZE (if X, then Y, because Z) → EXPERIMENT (real probe/action) → ANALYZE → LEARN (record in Memora `root/skill/research` or project branch). When stuck on a plateau, return to RESEARCH from a new angle or primary source rather than repeating the same query.