Malware Adversarial Eval

---
name: malware-adversarial-eval
description: Evaluates the robustness of deep neural network malware classifiers against adversarial attacks. It probes whether an attacker can successfully misclassify malicious Android applications by adding a limited number of valid features to their manifest files. Use when the user wants to benchmark on DREBIN, or asks about evaluating this task. Reports misclassification rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1606.04435
  bibtex_key: grosse2016adversarial
  confidence: high
---

# malware-adversarial-eval

> Adversarial Perturbations Against Deep Neural Networks for Malware Classification — Grosse et al. (2016) (arXiv:1606.04435, 2016)

## What this evaluates

Evaluates the robustness of deep neural network malware classifiers against adversarial attacks. It probes whether an attacker can successfully misclassify malicious Android applications by adding a limited number of valid features to their manifest files.

## Datasets

- **DREBIN** — total 129013; splits: train (-1), test (-1)

## Metrics

- `misclassification rate` **(primary)** — range: percent
  - Percentage of malware samples that are correctly classified initially but misclassified after applying the adversarial crafting algorithm.
- `accuracy` — range: percent
  - Overall percentage of correctly classified applications (malware and benign) on the dataset.
- `false negative rate` — range: percent
  - Percentage of actual malware samples incorrectly classified as benign.
- `false positive rate` — range: percent
  - Percentage of actual benign samples incorrectly classified as malware.
- `distortion` — range: other
  - Average number of features added to an application during adversarial crafting.

## Input / output format

**Input**: Binary indicator vector X ∈ {0,1}^M representing static features of an Android application (permissions, API calls, components, intents).

**Output**: Binary classification label (malware or benign).

## Scoring recipe

```python
def compute_misclassification_rate(correctly_classified_malware, model, craft_adversarial):
    misclassified = 0
    for sample in correctly_classified_malware:
        adv_sample = craft_adversarial(sample, max_mods=20)
        if model.predict(adv_sample) != 'malware':
            misclassified += 1
    return misclassified / len(correctly_classified_malware)
```

## Common pitfalls

- Perturbations must only modify features representable in AndroidManifest.xml to maintain app functionality.
- A strict distortion bound of k=20 added features is enforced, limiting the attack's magnitude.
- Training malware ratio (MWR) heavily impacts the trade-off between detection accuracy and adversarial robustness.

## Evidence (verbatim from paper)

> As a measure of success, we consider the misclassification rate achieved by the adversarial crafting algorithm. The misclassification rate is defined by the percentage of malware samples that are misclassified after applying the adversarial crafting algorithm, but were correctly classified before that.

## Citation

```bibtex
@misc{grosse2016adversarial,
  title={Adversarial Perturbations Against Deep Neural Networks for Malware Classification},
  author={Grosse et al. (2016)},
  year={2016},
  note={arXiv:1606.04435}
}
```

- arXiv: 1606.04435