1541 Developing 551cf265

# Developer Guide

## Architecture Documentation

- **[Architecture Overview](architecture/OVERVIEW.md)** - System components, research flow, and module responsibilities
- **[Database Schema](architecture/DATABASE_SCHEMA.md)** - Database models and relationships
- **[Extension Guide](developing/EXTENDING.md)** - How to add custom search engines, strategies, and LLM providers
- **[Testing and CI](CI_CD_INFRASTRUCTURE.md)** - GitHub Actions workflows, pre-commit hooks, and security scanning

## Development Setup

### Prerequisites
- **Python**: 3.11+
- **Node.js**: 24.0.0+
- **Docker**: Latest version (for production builds)
- **PDM**: Python package manager
- **SQLCipher**: Required for encrypted database support. See [SQLCIPHER_INSTALL.md](SQLCIPHER_INSTALL.md) for platform-specific instructions.

### Initial Setup

1. **Clone and Prepare Environment**
   ```bash
   git clone git@github.com:LearningCircuit/local-deep-research.git
   # Or via HTTPS: https://github.com/LearningCircuit/local-deep-research.git
   cd local-deep-research
   ```

2. **Backend Setup**
   ```bash
   # Create and activate virtual environment
   python -m venv .venv
   source .venv/bin/activate

   # Install dependencies
   pip install pdm
   pdm install --no-self
   ```

3. **Frontend Setup**
   ```bash
   # Install frontend dependencies
   npm install
   ```

4. **Enable Git Hooks**
   ```bash
   # Install pre-commit hooks
   pre-commit install
   pre-commit install-hooks
   ```
   We use the `pre-commit` framework to manage git hooks. This repository includes both standard checks (ruff, eslint, gitleaks) and custom local checks located in `.pre-commit-hooks/`.

## Running the Application

### Development Mode

1. **Start the Backend**
   ```bash
   source .venv/bin/activate

   # Option A: Using the installed entry point
   ldr-web

   # Option B: Using Python module directly
   python -m local_deep_research.web.app
   ```

2. **Start the Frontend** (in a new terminal)
   ```bash
   npm run dev
   ```
   Access the app at `http://localhost:5173`.

### Development Environment Variables

For local development and testing, you may want to configure these environment variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `LDR_DATA_DIR` | Platform default | Override data/database storage location |
| `LDR_BOOTSTRAP_ALLOW_UNENCRYPTED` | `false` | Allow unencrypted database (dev only!) |
| `CI` or `TESTING` | unset | Enables testing mode (bypasses some security checks) |

> **Warning**: Never set `LDR_BOOTSTRAP_ALLOW_UNENCRYPTED=true` in production. User data will be stored unencrypted.

### Docker (Production-like)

To build and run the entire stack in Docker:

```bash
docker build -t localdeepresearch/local-deep-research:dev .
docker run -p 5000:5000 -e LDR_DATA_DIR=/data -v ldr_data:/data localdeepresearch/local-deep-research:dev
```

## Building

### Building a Package

To build a wheel and source distribution, simply run `pdm build`.

### Building Frontend Assets

If you're developing from source and want to use the Web UI, you need to build the frontend assets:

```bash
npm install
npm run build
```

This builds the Vite frontend into `src/local_deep_research/web/static/dist/`.

> **Note:** pip users don't need this step - pre-built assets are included in the PyPI package.

### Dependency Lockfile Management

This project uses **PDM** with `pdm.lock` to pin exact dependency versions. If you see this warning during Docker builds:

```
WARNING: Lockfile hash doesn't match pyproject.toml, packages may be outdated
```

It means `pyproject.toml` has changed but `pdm.lock` hasn't been regenerated.

**How to fix:**
```bash
pdm lock
```

Always commit `pdm.lock` alongside `pyproject.toml` changes to ensure reproducible builds.

## Testing

### Backend Tests (Pytest)

We support two modes of backend testing:

#### 1. Isolated Testing (No Server Required)
This is the default and recommended way to run unit and integration tests. It uses Flask's `test_client` to mock the server.

**How to run:**
```bash
source .venv/bin/activate
# Run all isolated tests
pytest tests/

# Run specific API or Auth tests
pytest tests/api_tests/
pytest tests/auth_tests/
```

#### 2. Live System Testing (Requires Running Server)
These tests make real HTTP requests to a running application instance.

**Prerequisites:**
1. Start the backend server in one terminal: `ldr-web` (or via Docker).
2. Run the live test scripts in another terminal.

**How to run:**
```bash
python tests/ui_tests/test_simple_research_api.py
```

### Frontend & E2E Tests (Puppeteer)
We use `Puppeteer` for UI and End-to-End testing.

**Prerequisites:**
1. Navigate to the tests directory: `cd tests`
2. Install test dependencies: `npm install`

**How to run:**
1. **Ensure the application is running** (locally on port 5000 or via Docker).
2. Run the test suite:
   ```bash
   # Run all UI tests
   node tests/ui_tests/run_all_ui_tests.js

   # Run specific test
   node tests/ui_tests/test_simple_auth.js
   ```

## Database Management

### Backup & Restore
Before testing changes, you may wish to backup your Local Deep Research data volume.

**Create a Backup**
```bash
docker run --rm \
  -v ldr_data:/from \
  -v ldr_data-backup:/to \
  debian:latest \
  bash -c "cd /from ; tar -cf - . | (cd /to ; tar -xpf -)"
```

**Restore from Backup**

*Warning: This overwrites existing data*
```bash
docker run --rm \
  -v ldr_data:/target \
  -v ldr_data-backup:/source \
  debian:latest \
  bash -c "rm -rf /target/* /target/.[!.]* ; \
           cd /source ; tar -cf - . | (cd /target ; tar -xpf -)"
```

## Troubleshooting

### SQLCipher Issues

See [SQLCIPHER_INSTALL.md](SQLCIPHER_INSTALL.md#troubleshooting) for SQLCipher-related errors.

### Permission Denied on Docker Volume

**Error:** `PermissionError: [Errno 13] Permission denied: '/app/.config/...'`

**Solution:** The volume may have been created with different ownership. Reset it:
```bash
docker volume rm ldr_data
# Re-run the container to create a fresh volume
```

### Session Lost After Server Restart

**Cause:** The application's secret key is used to sign session cookies.

**Solution:** The secret key is automatically generated on first run and persisted to a `.secret_key` file in your data directory (controlled by `LDR_DATA_DIR`). Sessions are only lost if this file is deleted. If using Docker, ensure your data volume (`ldr_data:/data`) is persistent.

### NoSettingsContextError in Background Threads

**Error:** `NoSettingsContextError: No settings context available`

**Cause:** Background threads don't have access to the Flask request context.

**Solution:** This is handled automatically by the application. If you encounter this during development, ensure your LLM settings are configured via the web UI settings page. For CI/testing environments, the `LDR_TESTING_USE_FALLBACK_LLM=true` environment variable enables a mock LLM fallback.

### PDM Lockfile Out of Sync

**Error:** `Lockfile hash doesn't match pyproject.toml`

**Solution:**
```bash
pdm lock
git add pdm.lock
```