1317 Cli Tools 2fb9743f

# CLI Tools Reference

Local Deep Research includes command-line tools for benchmarking and rate limit management.

## Table of Contents

- [Benchmarking CLI](#benchmarking-cli)
- [Rate Limiting CLI](#rate-limiting-cli)

---

## Benchmarking CLI

Run benchmarks to evaluate search quality and compare configurations.

### Basic Usage

```bash
python -m local_deep_research.benchmarks.cli <command> [options]
```

### Commands

#### `simpleqa` - Run SimpleQA Benchmark

Tests factual question answering accuracy.

```bash
python -m local_deep_research.benchmarks.cli simpleqa [options]
```

**Options:**

| Option | Default | Description |
|--------|---------|-------------|
| `--examples` | 100 | Number of questions to test |
| `--iterations` | 3 | Search iterations per question |
| `--questions` | 3 | Questions per iteration |
| `--search-tool` | searxng | Search engine to use |
| `--search-strategy` | source_based | Strategy (source_based, standard, rapid, parallel, iterdrag) |
| `--search-model` | (default) | LLM model for research |
| `--search-provider` | (default) | LLM provider |
| `--eval-model` | (default) | Model for answer evaluation |
| `--eval-provider` | (default) | Provider for evaluation |
| `--output-dir` | ~/.local-deep-research/benchmark_results | Results directory |
| `--human-eval` | false | Use human evaluation |
| `--no-eval` | false | Skip evaluation phase |
| `--custom-dataset` | - | Path to custom dataset |

**Example:**

```bash
# Run 50 examples with Ollama
python -m local_deep_research.benchmarks.cli simpleqa \
  --examples 50 \
  --search-provider ollama \
  --search-model llama3.2
```

#### `browsecomp` - Run BrowseComp Benchmark

Tests complex reasoning and multi-step research.

```bash
python -m local_deep_research.benchmarks.cli browsecomp [options]
```

Same options as `simpleqa`.

**Example:**

```bash
# Run BrowseComp with focused-iteration strategy
python -m local_deep_research.benchmarks.cli browsecomp \
  --examples 20 \
  --search-strategy iterdrag \
  --iterations 5
```

#### `compare` - Compare Configurations

Compare multiple search configurations on the same dataset.

```bash
python -m local_deep_research.benchmarks.cli compare [options]
```

**Options:**

| Option | Default | Description |
|--------|---------|-------------|
| `--dataset` | simpleqa | Dataset to use (simpleqa, browsecomp) |
| `--examples` | 20 | Examples per configuration |
| `--output-dir` | ~/.local-deep-research/benchmark_results/comparison | Results directory |

**Example:**

```bash
# Compare configurations
python -m local_deep_research.benchmarks.cli compare \
  --dataset simpleqa \
  --examples 30
```

#### `list` - List Available Benchmarks

```bash
python -m local_deep_research.benchmarks.cli list
```

Shows available benchmark datasets and their descriptions.

---

## Rate Limiting CLI

Monitor and manage the adaptive rate limiting system.

### Basic Usage

```bash
python -m local_deep_research.web_search_engines.rate_limiting.cli <command> [options]
```

### Commands

#### `status` - Show Rate Limit Statistics

View current rate limit data for search engines.

```bash
# All engines
python -m local_deep_research.web_search_engines.rate_limiting.cli status

# Specific engine
python -m local_deep_research.web_search_engines.rate_limiting.cli status --engine DuckDuckGoSearchEngine
```

**Output columns:**

| Column | Description |
|--------|-------------|
| Engine | Search engine name |
| Base Wait | Current wait time in seconds |
| Range | Min-max wait times |
| Success | Success rate percentage |
| Attempts | Total request attempts |
| Updated | Last update timestamp |

**Example output:**

```
Rate Limit Statistics:
--------------------------------------------------------------------------------
Engine               Base Wait    Range                Success    Attempts   Updated
--------------------------------------------------------------------------------
DuckDuckGoSearchEngine 2.50        1.0s - 5.0s         95.2%      150        12-26 14:30
ArXivSearchEngine      0.50        0.5s - 1.0s         99.8%      85         12-26 12:15
```

#### `reset` - Reset Engine Rate Limits

Clear learned rate limit data for an engine.

```bash
python -m local_deep_research.web_search_engines.rate_limiting.cli reset --engine <engine_name>
```

**Example:**

```bash
# Reset DuckDuckGo rate limits
python -m local_deep_research.web_search_engines.rate_limiting.cli reset --engine DuckDuckGoSearchEngine
```

Use this when:
- Rate limits are too conservative
- After API changes
- When switching environments

#### `export` - Export Rate Limit Data

Export rate limit statistics in various formats.

```bash
# Table format (default)
python -m local_deep_research.web_search_engines.rate_limiting.cli export

# CSV format
python -m local_deep_research.web_search_engines.rate_limiting.cli export --format csv

# JSON format
python -m local_deep_research.web_search_engines.rate_limiting.cli export --format json
```

**Formats:**

| Format | Use Case |
|--------|----------|
| `table` | Human-readable display |
| `csv` | Spreadsheet import |
| `json` | Programmatic processing |

**Example CSV output:**

```csv
engine_type,base_wait_seconds,min_wait_seconds,max_wait_seconds,last_updated,total_attempts,success_rate
DuckDuckGoSearchEngine,2.5,1.0,5.0,1703612400,150,0.952
```

#### `cleanup` - Remove Old Data

Clean up rate limit data older than a specified number of days.

```bash
python -m local_deep_research.web_search_engines.rate_limiting.cli cleanup --days <days>
```

**Example:**

```bash
# Remove data older than 30 days
python -m local_deep_research.web_search_engines.rate_limiting.cli cleanup --days 30

# Remove data older than 7 days
python -m local_deep_research.web_search_engines.rate_limiting.cli cleanup --days 7
```

---

## Common Engine Names

When using the rate limiting CLI, use these engine class names:

| Engine | Class Name |
|--------|------------|
| DuckDuckGo | `DuckDuckGoSearchEngine` |
| SearXNG | `SearXNGSearchEngine` |
| Brave | `BraveSearchEngine` |
| arXiv | `ArXivSearchEngine` |
| PubMed | `PubMedSearchEngine` |
| Semantic Scholar | `SemanticScholarSearchEngine` |
| Wikipedia | `WikipediaSearchEngine` |
| GitHub | `GitHubSearchEngine` |

---

## Troubleshooting

### Benchmark Not Starting

- Verify LLM provider is configured
- Check search engine is available
- Ensure sufficient disk space for results

### Rate Limit Data Missing

- Run some searches first to generate data
- Check database file exists
- Try `status` without `--engine` flag

### Export Permission Error

- Check write permissions on output directory
- Use a different output directory

---

## See Also

- [Architecture Overview](architecture/OVERVIEW.md) - System architecture
- [Troubleshooting](troubleshooting.md) - Common issues
- [BENCHMARKING.md](BENCHMARKING.md) - Detailed benchmark documentation