Surgery Mapping Eval

---
name: surgery-mapping-eval
description: Evaluates the correctness and efficiency of gradient-based versus boolean logic-based methods for identifying feature-parameter interactions and transferring trained weights when new features are added to a reinforcement learning model. It measures how well each mapping technique preserves model performance and computational speed during architectural surgery. Use when the user has predictions and gold and needs to compute Interactions Found.
metadata:
  skill_kind: metric
  source_arxiv: 1912.06719
  bibtex_key: raiman2019neural
  confidence: high
---

# surgery-mapping-eval

> Neural Network Surgery with Sets — Raiman et al. (2019) (arXiv:1912.06719, 2019)

## What this evaluates

Evaluates the correctness and efficiency of gradient-based versus boolean logic-based methods for identifying feature-parameter interactions and transferring trained weights when new features are added to a reinforcement learning model. It measures how well each mapping technique preserves model performance and computational speed during architectural surgery.

## Datasets

- **OpenAI Five (OG model)** — total ?; splits: test (-1)

## Metrics

- `Interactions Found` **(primary)** — range: percent
  - Percentage of total feature-parameter interactions detected between the original and modified model versions.
- `Params Transferred` — range: percent
  - Percentage of the original model’s parameters successfully transferred to the new model architecture under specific initialization and zero-gradient handling.
- `Time (secs)` — range: seconds
  - Wall-clock time required to compute the feature-parameter interaction map using the specified mapping technique.

## Input / output format

**Input**: Original model parameters and architecture, modified model architecture with added features (12 per-hero features), and forward pass data to compute gradients or boolean logic traces.

**Output**: A computed feature-parameter interaction map, a set of transferred parameters, and the computation time in seconds.

## Scoring recipe

```python
def evaluate_surgery(old_model, new_model, features, init_method, zero_grad_handling):
    interactions_detected = count_interactions(old_model, new_model, features, method)
    interactions_total = total_possible_interactions
    interactions_pct = (interactions_detected / interactions_total) * 100

    params_transferred = count_transferred_weights(old_model.params, new_model.params, init_method, zero_grad_handling)
    params_total = len(old_model.params)
    params_pct = (params_transferred / params_total) * 100

    time_taken = measure_wall_clock_time(compute_mapping)
    return interactions_pct, params_pct, time_taken
```

## Common pitfalls

- Random initialization alone masks interactions (49.42%) and prevents optimal transfer; proper initialization (random positive) and zero-gradient function replacement are required for 100% detection.
- Boolean logic mapping is agnostic to initialization and zero-gradient functions, whereas gradient mapping requires specific handling to avoid masking interactions.
- Time measurements are hardware-dependent (reported on a 2.9 GHz Intel Core i7) and should not be directly compared across different machines without normalization.

## Evidence (verbatim from paper)

> In Table 3 we report the percentage of total feature-parameter interactions detected, as well as the percentage of the original model’s parameters that can be transferred to the new model under different initializations and zero-gradient function replacements. We find that random positive initialization and zero-gradient function replacement achieves the highest number of transferred parameters, and detects all feature-parameter interactions.

## Citation

```bibtex
@misc{raiman2019neural,
  title={Neural Network Surgery with Sets},
  author={Raiman et al. (2019)},
  year={2019},
  note={arXiv:1912.06719}
}
```

- arXiv: 1912.06719