Deep Research

---
name: deep-research
description: Perform comprehensive web research using local SearXNG. Iteratively searches, fetches content, and synthesizes a report with citations. Use for complex research questions requiring multiple sources.
homepage: https://github.com/romancircus/searxng-deep-research
metadata: {"clawdbot":{"emoji":"🔬","requires":{"bins":["python3"],"python":["aiohttp","beautifulsoup4"]},"install":[{"id":"python-deps","kind":"pip","packages":["aiohttp","beautifulsoup4"],"label":"Install Python dependencies"}]}}
---

# Deep Research

Performs iterative web research via local SearXNG (Google-free, VPN-routed).

## Quick Usage

```bash
python3 ~/.clawdbot/skills/deep-research/deep_research.py "your research question"
```

Or use the CLI wrapper:
```bash
deep-research "what are the best practices for kubernetes security in 2026"
```

## How It Works

1. **Iterative Search** - Up to 5 iterations with query refinement
2. **Content Fetching** - Scrapes full page content from valid URLs
3. **Deduplication** - Tracks seen URLs to avoid duplicates
4. **Report Generation** - Produces markdown with citations

## Algorithm

```
for iteration in 1..5:
    query = refine_query(original_query, iteration)
    results = search_searxng(query, offset=iteration * 10)

    for result in results:
        if url not in seen_urls and domain not in ignored:
            content = fetch_and_scrape(url)
            add_to_findings(title, url, content)

    if sufficient_results:
        break

generate_markdown_report(findings, citations)
```

## Query Refinement

Each iteration adds context terms:
- Iteration 1: Original query
- Iteration 2: + "detailed analysis"
- Iteration 3: + "comprehensive guide"
- Iteration 4: + "in-depth review"
- Iteration 5: + "research findings"

## Configuration

Edit `~/.clawdbot/skills/deep-research/deep_research.py`:

```python
SEARXNG_URL = "http://localhost:8888"  # Your SearXNG instance
MAX_ITERATIONS = 5                      # Search iterations
RESULTS_PER_PAGE = 10                   # Results per iteration
PAGE_CONTENT_LIMIT = 2000               # Max words per source
REQUEST_TIMEOUT = 20                    # Fetch timeout (seconds)
```

## Ignored Domains

Social media and low-value domains are excluded:
- youtube.com, facebook.com, twitter.com
- instagram.com, tiktok.com, pinterest.com, linkedin.com

Edit `IGNORED_DOMAINS` list to customize.

## Output Format

```markdown
# Deep Research Report

**Query:** your research question
**Date:** 2026-01-27 15:30
**Sources:** 8

---

## Research Findings

### [1] Article Title
**Source:** https://example.com/article

Content preview from the article...

### [2] Another Source
...

---

## Sources

1. [Article Title](https://example.com/article)
2. [Another Source](https://example.com/other)
```

## Privacy Features

- **No Google/Bing** - Uses privacy-respecting engines only
- **VPN Routed** - Traffic goes through Tailscale/Mullvad
- **Local Processing** - All synthesis happens locally
- **No API Keys** - Self-hosted SearXNG, no external dependencies

## Requirements

- Python 3.8+
- Local SearXNG instance at port 8888
- Python packages: `aiohttp`, `beautifulsoup4`

Install dependencies:
```bash
pip install aiohttp beautifulsoup4
```

## Examples

Research a technical topic:
```bash
python3 ~/.clawdbot/skills/deep-research/deep_research.py "rust async runtime comparison tokio vs async-std 2026"
```

Investigate a concept:
```bash
python3 ~/.clawdbot/skills/deep-research/deep_research.py "zero knowledge proofs practical applications"
```

Compare technologies:
```bash
python3 ~/.clawdbot/skills/deep-research/deep_research.py "comparing vector databases pinecone vs milvus vs qdrant"
```

## Troubleshooting

**No results found:**
- Check SearXNG is running: `curl http://localhost:8888`
- Verify query isn't too specific
- Try broader search terms

**Slow performance:**
- Reduce MAX_ITERATIONS
- Decrease RESULTS_PER_PAGE
- Some sites have rate limiting

**Content not extracted:**
- Site may require JavaScript (not supported)
- Try the URL directly in browser
- Content may be behind paywall