Malimg Eval

---
name: malimg-eval
description: Evaluates the ability of deep learning models to classify malware families from their visual representations (malware images). It probes feature extraction robustness and classification accuracy on an imbalanced dataset. Use when the user wants to benchmark on MalImg, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2107.04008
  bibtex_key: asam2021malware
  confidence: high
---

# malimg-eval

> Malware Classification Using Deep Boosted Learning — Muhammad Asam et al. (2021) (arXiv:2107.04008, 2021)

## What this evaluates

Evaluates the ability of deep learning models to classify malware families from their visual representations (malware images). It probes feature extraction robustness and classification accuracy on an imbalanced dataset.

## Datasets

- **MalImg** — total ?; splits: train (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - The proportion of correctly classified samples out of the total number of samples.
- `F1-Score` — range: [0, 1]
  - The harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall).
- `Precision` — range: [0, 1]
  - The ratio of true positive predictions to the total number of positive predictions.
- `Recall` — range: [0, 1]
  - The ratio of true positive predictions to the total number of actual positives.

## Input / output format

**Input**: Grayscale or RGB images of malware binaries (visual representations).

**Output**: Discrete malware family/class label.

## Scoring recipe

```python
accuracy = sum(pred == gold) / len(gold)
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
```

## Common pitfalls

- Different papers use different train/test splits (e.g., 70-30 vs 60-40), making direct accuracy comparisons misleading.
- The dataset is imbalanced; relying solely on Accuracy can mask poor performance on minority classes, which is why F1-score is recommended as the standard measure.

## Evidence (verbatim from paper)

> In order to make our proposed malware classification framework more robust, we implemented the architectures in the data distribution of 60-40 ratio. These experiments are performed using (i) softmax probability-based Malware classification, (ii) deep feature and ML-based Malware classification and (iii) our proposed malware classification approach (DFS-MC). These malware classification models are implemented using training from scratch for MalImg data set in the first place and then using pre-trained models for ImageNet. Performance measures like accuracy and F1-score of training from scratch and TL-based implementation are recorded.

## Citation

```bibtex
@misc{asam2021malware,
  title={Malware Classification Using Deep Boosted Learning},
  author={Muhammad Asam et al. (2021)},
  year={2021},
  note={arXiv:2107.04008}
}
```

- arXiv: 2107.04008