Malimg Classification Eval

---
name: malimg-classification-eval
description: This benchmark evaluates the ability of machine learning models to classify malware binaries by converting them into grayscale images and predicting their specific family among 25 categories. It probes classification accuracy and computational efficiency across different neural network architectures. Use when the user wants to benchmark on Malimg, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.13820
  bibtex_key: jain2021cnneelm
  confidence: high
---

# malimg-classification-eval

> CNN vs ELM for Image-Based Malware Classification — Jain et al. (2021) (arXiv:2103.13820, 2021)

## What this evaluates

This benchmark evaluates the ability of machine learning models to classify malware binaries by converting them into grayscale images and predicting their specific family among 25 categories. It probes classification accuracy and computational efficiency across different neural network architectures.

## Datasets

- **Malimg** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly classified samples out of the total number of samples. Calculated as the number of correct predictions divided by the total number of predictions.

## Input / output format

**Input**: Grayscale images of malware binaries generated via byte-to-pixel mapping (sizes: 32×32, 64×64, or 128×128 pixels), or 1D feature vectors of fixed length (512 or 1024) for ELM experiments.

**Output**: A single class label corresponding to one of the 25 malware families in the Malimg dataset.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- The Malimg dataset is highly imbalanced across its 25 classes, which can bias standard accuracy metrics toward majority classes.
- ELM results are averaged over 50 independent runs due to random weight initialization, so single-run evaluations will show high variance.
- Input image resolution significantly affects CNN accuracy but has minimal impact on 1D vector-based ELM performance.

## Evidence (verbatim from paper)

> For each combination of parameters, we train and test 50 ELMs, and evaluate the performance in terms of the average accuracy.

## Citation

```bibtex
@misc{jain2021cnneelm,
  title={CNN vs ELM for Image-Based Malware Classification},
  author={Jain et al. (2021)},
  year={2021},
  note={arXiv:2103.13820}
}
```

- arXiv: 2103.13820