Spike Nvpt Robust Visual Prompt

---
name: spike-nvpt-robust-visual-prompt
description: "Spike-NVPT: Spiking Neuron-based Robust Visual Prompt Tuning. Uses Integrate-and-Fire (IF) mechanism as signal filtering layer with spiking discretization units to generate sparse binary prompts for robust vision model fine-tuning. Zero inference overhead — spiking components are training-only. First work using spiking neurons to fine-tune ANN visual models. Use when: robust visual prompt tuning, adversarial robustness, spiking neuron applications in CV, parameter-efficient fine-tuning, ANN-SNN hybrid training, corruption-resistant vision models."
version: 1.0.0
---

# Spike-NVPT: Spiking Neuron-based Robust Visual Prompt Tuning

## Core Insight

Spiking neurons' natural signal filtering properties (Integrate-and-Fire mechanism) can be repurposed as robust prompt generators for vision models. The key insight: IF neurons act as low-pass filters that suppress high-frequency noise/adversarial perturbations while preserving semantic content.

## Architecture

### 1. Spiking Signal Filtering Layer (Training-Only)
```python
import torch
import torch.nn as nn

class SpikingFilterLayer(nn.Module):
    """IF-neuron based signal filter for robust prompt generation."""
    def __init__(self, threshold=1.0, decay=0.5):
        super().__init__()
        self.threshold = threshold
        self.decay = decay
    
    def forward(self, x):
        # x: input feature map
        membrane = torch.zeros_like(x)
        spikes = torch.zeros_like(x)
        
        # Integrate phase
        membrane = self.decay * membrane + x
        
        # Fire phase — binary discretization
        spikes = (membrane >= self.threshold).float()
        
        # Reset membrane after spike
        membrane = membrane * (1 - spikes)
        
        return spikes  # Sparse binary prompt
```

### 2. Spiking Discretization Unit
```python
class SpikingDiscretizationUnit(nn.Module):
    """Converts continuous features to sparse binary prompts."""
    def __init__(self, embed_dim, prompt_length=10):
        super().__init__()
        self.linear = nn.Linear(embed_dim, prompt_length)
        self.if_neuron = SpikingFilterLayer(threshold=0.5)
    
    def forward(self, x):
        projected = self.linear(x.mean(dim=1))  # Pool spatial dims
        binary_prompt = self.if_neuron(projected)
        return binary_prompt  # Sparse binary: 0s and 1s only
```

### 3. Training Pipeline
```python
class SpikeNVPT(nn.Module):
    def __init__(self, backbone, prompt_length=10, embed_dim=768):
        super().__init__()
        self.backbone = backbone
        self.prompt_generator = SpikingDiscretizationUnit(embed_dim, prompt_length)
        
        # Freeze backbone
        for param in self.backbone.parameters():
            param.requires_grad = False
    
    def forward(self, x):
        features = self.backbone.forward_features(x)
        
        # Generate sparse binary prompt via spiking
        prompt = self.prompt_generator(features)
        
        # Inject prompt into backbone (no extra inference cost at test time)
        output = self.backbone.forward_head(features, prompt=prompt)
        return output
```

## Key Properties

1. **Zero inference overhead**: Spiking components produce binary prompts — can be pre-computed or removed at deployment
2. **Robustness**: IF mechanism filters adversarial perturbations naturally (low-pass filtering)
3. **Sparsity**: Binary prompts reduce computation vs continuous soft prompts
4. **ANN-SNN hybrid**: First work using SNN components to fine-tune conventional ANN vision models

## Results (arXiv:2604.18284, Apr 2026)
- **Up to 11.2% robustness improvement** over baseline visual prompt tuning
- Tested on ImageNet-C and multiple corruption benchmarks
- Authors: Qiugang Zhan + 6 co-authors | Categories: cs.CV

## Activation Keywords
- spike-nvpt
- spiking visual prompt
- robust prompt tuning
- IF neuron prompt
- spiking discretization
- ANN-SNN hybrid fine-tuning
- adversarial robustness prompting
- binary prompt tuning

## References
- Zhan et al., "Spike-NVPT: Spiking Neuron-based Robust Visual Prompt Tuning", arXiv:2604.18284, Apr 2026