Malware Static Reachability Eval

---
name: malware-static-reachability-eval
description: Evaluates a static analysis tool's ability to detect malware by extracting system call data flow trees and matching them against a learned tree automaton. It probes the model's capability to generalize semantic malware signatures from a small training set to a larger, unseen test set while avoiding false positives on benign software. Use when the user wants to benchmark on VX Heavens & Windows XP Benign, or asks about evaluating this task. Reports detection_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1312.4814
  bibtex_key: macedo2013mining
  confidence: high
---

# malware-static-reachability-eval

> Mining Malware Specifications through Static Reachability Analysis — Macedo et al. (2013) (arXiv:1312.4814, 2013)

## What this evaluates

Evaluates a static analysis tool's ability to detect malware by extracting system call data flow trees and matching them against a learned tree automaton. It probes the model's capability to generalize semantic malware signatures from a small training set to a larger, unseen test set while avoiding false positives on benign software.

## Datasets

- **VX Heavens & Windows XP Benign** — total 1426; splits: train (193), test_malware (983), test_benign (250)

## Metrics

- `detection_rate` **(primary)** — range: [0, 1]
  - True Positives / Total Malware Samples in Test Set. Reports the fraction of unseen malware binaries correctly identified as malicious.
- `false_positive_rate` — range: [0, 1]
  - False Positives / Total Benign Samples in Test Set. Reports the fraction of benign Windows XP binaries incorrectly flagged as malware.

## Input / output format

**Input**: Windows executable binary files (PE format).

**Output**: Binary classification label: 'malware' if any extracted System Call Data Flow Tree (SCDT) subtree is recognized by the learned tree automaton, otherwise 'benign'.

## Scoring recipe

```python
def evaluate(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'malware' and g == 'malware')
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'malware' and g == 'benign')
    total_malware = sum(1 for g in gold_labels if g == 'malware')
    total_benign = sum(1 for g in gold_labels if g == 'benign')
    detection_rate = tp / total_malware if total_malware > 0 else 0.0
    fpr = fp / total_benign if total_benign > 0 else 0.0
    return {'detection_rate': detection_rate, 'false_positive_rate': fpr}
```

## Common pitfalls

- The train/test split is arbitrary and not stratified by malware family, which may affect generalization claims across different threat types.
- Static analysis relies on external tools (Jakstab, IDA Pro) and approximations for stack values, which can lead to overapproximated trees and missed detections if the oracle is imprecise.
- Detection is binary and based on subtree matching; benign programs with similar system calls but different control/data flow may be misclassified if the automaton is too permissive.

## Evidence (verbatim from paper)

> We were able to detect 983 malware files using the malicious trees inferred from 193 malware files, and show that benign programs are benign, thus a 0% false positive rate.

## Citation

```bibtex
@misc{macedo2013mining,
  title={Mining Malware Specifications through Static Reachability Analysis},
  author={Macedo et al. (2013)},
  year={2013},
  note={arXiv:1312.4814}
}
```

- arXiv: 1312.4814