Malware Family Grouping Eval

---
name: malware-family-grouping-eval
description: Evaluates the capability of anti-malware detection engines and behavior profiling methods to correctly group malware variants into their respective families based on runtime Windows API call sequences and parameters. Use when the user wants to benchmark on 40Bot, 419Mal, or asks about evaluating this task. Reports Pairwise Classification Score (PCS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1705.01697
  bibtex_key: hsiao2017vmi
  confidence: high
---

# malware-family-grouping-eval

> Virtual Machine Introspection Based Malware Behavior Profiling and Family Grouping — Hsiao et al. (2017) (arXiv:1705.01697, 2017)

## What this evaluates

Evaluates the capability of anti-malware detection engines and behavior profiling methods to correctly group malware variants into their respective families based on runtime Windows API call sequences and parameters.

## Datasets

- **40Bot** — total 40; splits: test (40)
- **419Mal** — total 272; splits: test (272)

## Metrics

- `Pairwise Classification Score (PCS)` **(primary)** — range: other
  - PCS_x = (1/m) * W_x * sum_{y=1}^m P_x(y), where W_x is the detection rate of engine E_x, and P_x(y) is the approval rate from engine E_y based on pairwise family agreement. Engines receive positive points when they agree on whether a malware pair belongs to the same family, and negative otherwise.
- `Jaccard distance` — range: [0, 1]
  - Measures behavioral similarity between malware profiles based on shared Windows API calls and parameters. Calculated as the ratio of common API/parameter pairs to the total union of API/parameter pairs across two profiles.

## Input / output format

**Input**: Windows API call sequences with parameters and return values captured via Virtual Machine Introspection over a 300-second runtime window.

**Output**: Malware family assignment labels per variant, or a behavior profile vector used for hierarchical clustering.

## Scoring recipe

```python
def compute_PCS(engine_x, all_engines, malwares):
    m = len(all_engines)
    W_x = len([m for m in malwares if engine_x.detects(m)]) / len(malwares)
    total_approval = 0
    for engine_y in all_engines:
        agreements = 0
        for i, j in combinations(malwares, 2):
            same_x = (engine_x.family(i) == engine_x.family(j)) and engine_x.family(i) is not None
            same_y = (engine_y.family(i) == engine_y.family(j)) and engine_y.family(i) is not None
            if same_x == same_y:
                agreements += 1
        total_approval += agreements / len(combinations(malwares, 2))
    return (1/m) * W_x * total_approval
```

## Common pitfalls

- The evaluation lacks ground-truth family labels, relying entirely on peer voting among commercial engines, which can propagate systematic naming biases.
- Using only API names without parameters/return values (no_par) fails to distinguish malware from benign programs, as they share high API overlap.
- Direct string comparison of engine family names is invalid due to inconsistent naming schemes across detection engines.

## Evidence (verbatim from paper)

> We propose a metric, Pairwise Classification Score (PCS), for the evaluation of goodness of detection engines against the proposed VMI-based behavior profiles while using them in family grouping. The intuition of PCS adopts the idea of crowd intelligence that a good detection engine should be acclaimed by other engines. Assume there are n malwares, denoted as M_i, and m detection engines, denoted as E_x. ... PCS_x = 1/m * W_x * sum_{y=1}^m P_x(y)

## Citation

```bibtex
@misc{hsiao2017vmi,
  title={Virtual Machine Introspection Based Malware Behavior Profiling and Family Grouping},
  author={Hsiao et al. (2017)},
  year={2017},
  note={arXiv:1705.01697}
}
```

- arXiv: 1705.01697