Neural Thickets Dense Task Experts

---
name: neural-thickets-dense-task-experts
title: "Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights"
version: 0.0.2
engine: skillxiv-v0.0.2-claude-opus-4.6
license: MIT
url: "https://arxiv.org/abs/2603.12228"
keywords: [Pretraining, Adaptation, Ensembling, Task Specialization, Model Landscape]
description: "Discover that large pretrained models have dense neighborhoods of task-specific experts—random weight perturbations improve performance. Use RandOpt: sample perturbations, select top performers, ensemble via voting for multi-task adaptation."
---

# Technique: RandOpt—Ensembling via Random Perturbations of Pretrained Weights

Traditional fine-tuning assumes isolated optimal solutions in the loss landscape. Neural Thickets reveals a different regime for large pretrained models: the neighborhood around pretrained weights contains **abundant diverse task-improving specialists**. This "thicket regime" enables a surprisingly simple approach: randomly perturb the pretrained weights, select performers, and ensemble via majority voting.

This emerges from the fundamental difference in loss landscapes between small models ("needle in haystack" sparsity) and large pretrained models (dense solution neighborhoods).

## Core Concept

RandOpt operates in two phases:

**Training Phase**: Generate N random Gaussian perturbations of pretrained weights θ' = θ + σ·ϵ, evaluate on validation set, select top-K performers.

**Inference Phase**: Generate predictions using only the K selected models, aggregate via majority voting.

The method requires no additional training—just sampling, selection, and ensemble voting. Yet it often outperforms standard fine-tuning by 5-15% on diverse tasks.

## Architecture Overview

- **Pretrained backbone**: Frozen weight source θ
- **Perturbation sampler**: Generates N random Gaussian variants
- **Performance evaluator**: Validation-based selection of top-K
- **Ensemble pool**: K model copies with different weights
- **Aggregator**: Majority voting for final predictions

## Implementation Steps

### Step 1: Generate Random Weight Perturbations

Sample Gaussian noise, create perturbed model copies, and evaluate on validation set.

```python
import torch
import numpy as np

class RandOptEnsemble:
    def __init__(self, base_model, perturbation_scale=0.1, k_selected=5):
        self.base_model = base_model
        self.perturbation_scale = perturbation_scale
        self.k_selected = k_selected
        self.selected_models = []
        self.base_weights = self.get_model_weights(base_model)

    def get_model_weights(self, model):
        """Extract flattened weights from model."""
        weights = []
        for param in model.parameters():
            weights.append(param.data.clone().flatten())
        return torch.cat(weights)

    def set_model_weights(self, model, weights):
        """Set model weights from flattened tensor."""
        offset = 0
        for param in model.parameters():
            param_size = param.numel()
            param.data = weights[offset:offset + param_size].reshape(param.shape)
            offset += param_size

    def generate_perturbed_models(self, num_perturbations=100):
        """
        Create N random perturbations of the base model.
        """
        perturbed_models = []

        for _ in range(num_perturbations):
            # Generate random Gaussian perturbation
            perturbation = torch.randn_like(self.base_weights) * self.perturbation_scale

            # Create perturbed weights
            perturbed_weights = self.base_weights + perturbation

            # Clone base model and set perturbed weights
            perturbed_model = self.clone_model(self.base_model)
            self.set_model_weights(perturbed_model, perturbed_weights)

            perturbed_models.append(perturbed_model)

        return perturbed_models

    def clone_model(self, model):
        """Deep clone a model."""
        import copy
        return copy.deepcopy(model)
```

### Step 2: Evaluate and Select Top-K Models

Benchmark perturbations on validation set, retain only the best performers.

```python
def select_top_k_models(
    perturbed_models,
    validation_data,
    task_metric,
    k=5,
    batch_size=32
):
    """
    Evaluate all perturbations, return top-k by validation metric.

    task_metric: function(outputs, targets) -> score
    """
    model_scores = []

    for idx, model in enumerate(perturbed_models):
        model.eval()

        total_score = 0
        num_batches = 0

        with torch.no_grad():
            for batch_idx, (inputs, targets) in enumerate(validation_data):
                if batch_idx * batch_size >= 1000:  # Use subset for speed
                    break

                outputs = model(inputs)
                score = task_metric(outputs, targets)

                total_score += score.item()
                num_batches += 1

        avg_score = total_score / num_batches
        model_scores.append((idx, avg_score, model))

    # Sort by performance
    model_scores.sort(key=lambda x: x[1], reverse=True)

    # Select top-k
    selected = model_scores[:k]

    return [model for _, _, model in selected]
```

### Step 3: Ensemble Inference via Majority Voting

For classification, aggregate predictions across ensemble members.

```python
class RandOptEnsembleInference:
    def __init__(self, selected_models):
        self.selected_models = selected_models

    def forward(self, inputs):
        """
        inputs: batch of examples
        returns: ensemble predictions via majority voting
        """
        batch_size = inputs.shape[0]
        num_models = len(self.selected_models)

        # Collect predictions from all models
        all_predictions = []

        for model in self.selected_models:
            model.eval()

            with torch.no_grad():
                outputs = model(inputs)

                # For classification: argmax to get class
                predictions = torch.argmax(outputs, dim=-1)  # (batch_size,)
                all_predictions.append(predictions)

        # Stack: (num_models, batch_size)
        all_predictions = torch.stack(all_predictions)

        # Majority voting
        ensemble_predictions = torch.mode(all_predictions, dim=0)[0]

        return ensemble_predictions

    def forward_with_confidence(self, inputs):
        """
        Also return confidence from voting agreement.
        """
        batch_size = inputs.shape[0]
        num_models = len(self.selected_models)

        all_predictions = []
        for model in self.selected_models:
            model.eval()
            with torch.no_grad():
                outputs = model(inputs)
                predictions = torch.argmax(outputs, dim=-1)
                all_predictions.append(predictions)

        all_predictions = torch.stack(all_predictions)
        ensemble_predictions = torch.mode(all_predictions, dim=0)[0]

        # Confidence: fraction voting for majority
        agreement = (all_predictions == ensemble_predictions.unsqueeze(0)).float()
        confidence = agreement.mean(dim=0)  # (batch_size,)

        return ensemble_predictions, confidence
```

### Step 4: Multi-Task Adaptation

Extend RandOpt to multi-task scenarios with shared perturbations.

```python
def randopt_multitask_adaptation(
    base_model,
    tasks,
    perturbation_scale=0.1,
    num_perturbations=100,
    k_per_task=5
):
    """
    Adapt base model to multiple tasks via perturbations.

    tasks: list of (task_name, validation_data, metric_fn)
    """
    # Generate shared perturbations
    base_weights = torch.cat([p.data.flatten() for p in base_model.parameters()])

    perturbed_models_all = []
    for _ in range(num_perturbations):
        perturbation = torch.randn_like(base_weights) * perturbation_scale
        perturbed_weights = base_weights + perturbation
        perturbed_models_all.append(perturbed_weights)

    # Per-task selection
    task_ensembles = {}

    for task_name, val_data, metric_fn in tasks:
        # Evaluate all perturbations on this task
        model_scores = []

        for weight_vec in perturbed_models_all:
            # Create model with these weights
            temp_model = create_model_with_weights(base_model, weight_vec)

            # Evaluate
            score = evaluate_model(temp_model, val_data, metric_fn)
            model_scores.append((score, weight_vec))

        # Select top-k for this task
        model_scores.sort(key=lambda x: x[0], reverse=True)
        top_weights = [w for _, w in model_scores[:k_per_task]]

        task_ensembles[task_name] = top_weights

    return task_ensembles
```

### Step 5: Practical Integration Example

End-to-end example showing RandOpt for adaptation.

```python
def adapt_model_with_randopt(
    base_model,
    validation_data,
    test_data,
    task_metric,
    num_perturbations=100,
    perturbation_scale=0.1,
    k_selected=5
):
    """
    Full RandOpt pipeline: perturb, select, ensemble.
    """
    # Step 1: Generate perturbations
    ensemble = RandOptEnsemble(base_model, perturbation_scale, k_selected)
    perturbed_models = ensemble.generate_perturbed_models(num_perturbations)

    # Step 2: Select top-k
    selected_models = select_top_k_models(
        perturbed_models,
        validation_data,
        task_metric,
        k=k_selected
    )

    print(f"Selected {len(selected_models)} specialist models from {num_perturbations} perturbations")

    # Step 3: Ensemble inference
    ensemble_model = RandOptEnsembleInference(selected_models)

    # Evaluate on test set
    total_correct = 0
    total_samples = 0

    with torch.no_grad():
        for inputs, targets in test_data:
            predictions, confidence = ensemble_model.forward_with_confidence(inputs)

            correct = (predictions == targets).sum().item()
            total_correct += correct
            total_samples += targets.shape[0]

    accuracy = total_correct / total_samples
    print(f"Test Accuracy: {accuracy:.4f}")

    return ensemble_model, accuracy
```

## Practical Guidance

**When to Use:**
- Adapting large pretrained models to new tasks
- Multi-task scenarios requiring diverse specialists
- Scenarios where ensemble voting aligns with task objectives
- Low-budget adaptation (no fine-tuning needed)

**When NOT to Use:**
- Single-task scenarios (use standard fine-tuning)
- Real-time inference where latency is critical (ensemble requires K forward passes)
- Tasks where diverse predictions hurt (require confidence-weighted outputs)
- Small models where "thicket regime" doesn't apply

**Hyperparameter Tuning:**
- **perturbation_scale σ**: 0.05-0.2; larger = more diversity, higher variance
- **num_perturbations**: 50-200; more thorough but slower
- **k_selected**: 3-10; balance diversity and computational cost
- **validation set size**: Use 10-20% of data for selection

**Common Pitfalls:**
- Using validation performance on held-out test set (causes overfitting to validation)
- Perturbation scale too large causing out-of-distribution behavior
- Insufficient perturbations missing good specialists
- Forgetting to freeze base model weights (no training, only sampling!)

## Reference

[Neural Thickets paper on arXiv](https://arxiv.org/abs/2603.12228)