Open Set Malware Eval

---
name: open-set-malware-eval
description: Evaluates a model's ability to classify malware into known families while simultaneously detecting instances belonging to novel, unseen families in an open-set scenario. Use when the user wants to benchmark on BIG 2015, Mailing, MAL-100, or asks about evaluating this task. Reports classification accuracy ($C_{Acc}$).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.01236
  bibtex_key: guo2023cnsnet
  confidence: high
---

# open-set-malware-eval

> CNS-Net: Conservative Novelty Synthesizing Network for Malware Recognition in an Open-set Scenario — Jingcai Guo et al. (2023) (arXiv:2305.01236, 2023)

## What this evaluates

Evaluates a model's ability to classify malware into known families while simultaneously detecting instances belonging to novel, unseen families in an open-set scenario.

## Datasets

- **BIG 2015** — total 10868; splits: train (6900), test (3968)
- **Mailing** — total 9339; splits: train (6400), test (2939)
- **MAL-100** — total 56481; splits: train (31523), test (24958)

## Metrics

- `classification accuracy ($C_{Acc}$)` **(primary)** — range: percent
  - Calculated as $N_{correct} / N_{instance}$, where $N_{correct}$ is the number of correctly classified known malware instances and $N_{instance}$ is the total number of testing known instances.
- `detection accuracy ($D_{Acc}$)` — range: percent
  - Calculated as $(TPR_{(K)} + TNR_{(U)}) / 2$, where $TPR_{(K)}$ is the true positive rate for known families and $TNR_{(U)}$ is the true negative rate for unknown families.

## Input / output format

**Input**: Malware instances represented as 25×25 grayscale images (or 1D vectors for some baselines) derived from 8 groups of raw characteristics including PE header info, COFF header, imported/exported functions, section info, byte histogram, byte-entropy histogram, and printable-string info.

**Output**: Predicted malware family label for known instances, and a binary known/unknown decision for detection.

## Scoring recipe

```python
def compute_C_Acc(preds, golds):
    correct = sum(1 for p, g in zip(preds, golds) if p == g)
    return correct / len(golds)

def compute_D_Acc(preds, golds, is_unknown):
    tp_k = sum(1 for p, g, u in zip(preds, golds, is_unknown) if p == g and not u)
    fn_k = sum(1 for p, g, u in zip(preds, golds, is_unknown) if p != g and not u)
    tpr_k = tp_k / (tp_k + fn_k) if (tp_k + fn_k) > 0 else 0
    tn_u = sum(1 for p, g, u in zip(preds, golds, is_unknown) if p == 'unknown' and u)
    fp_u = sum(1 for p, g, u in zip(preds, golds, is_unknown) if p != 'unknown' and u)
    tnr_u = tn_u / (tn_u + fp_u) if (tn_u + fp_u) > 0 else 0
    return (tpr_k + tnr_u) / 2
```

## Common pitfalls

- Data splitting is not standardized across baselines; some competitors use different train/test ratios or only report results on known families.
- The $D_{Acc}$ metric averages TPR for knowns and TNR for unknowns, which can mask severe imbalance in detection performance between known and unknown classes.
- Resizing malware feature vectors to fixed 25×25 or 32×32 images may discard fine-grained structural details present in the original 622-dimensional feature space.

## Evidence (verbatim from paper)

> The performance of open-set malware recognition can be evaluated by two parallel tasks including the multi-families classification and the unknown detection. The performance is evaluated by the classification accuracy $C_{Acc}$ defined as: $C_{Acc} = N_{correct} / N_{instance}$... The detection accuracy $D_{Acc}$ is then calculated as: $D_{Acc} = (TPR_{(K)} + TNR_{(U)}) / 2$.

## Citation

```bibtex
@misc{guo2023cnsnet,
  title={CNS-Net: Conservative Novelty Synthesizing Network for Malware Recognition in an Open-set Scenario},
  author={Jingcai Guo et al. (2023)},
  year={2023},
  note={arXiv:2305.01236}
}
```

- arXiv: 2305.01236