Malware Classification Eval

---
name: malware-classification-eval
description: This evaluation probes a model's ability to classify long-sequence binary and executable files into malware families or benign/malicious categories. It tests robustness to varying sequence lengths, compression formats, and real-world malware distribution characteristics compared to synthetic long-range benchmarks. Use when the user wants to benchmark on Kaggle (BIG 2015), Drebin, EMBER, LRA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.17978
  bibtex_key: alam2024holographic
  confidence: high
---

# malware-classification-eval

> Holographic Global Convolutional Networks for Long-Range Prediction Tasks in Malware Detection — Alam et al. (2024) (arXiv:2403.17978, 2024)

## What this evaluates

This evaluation probes a model's ability to classify long-sequence binary and executable files into malware families or benign/malicious categories. It tests robustness to varying sequence lengths, compression formats, and real-world malware distribution characteristics compared to synthetic long-range benchmarks.

## Datasets

- **Kaggle (BIG 2015)** — total 10868; splits: train (8694), test (2174)
- **Drebin** — total 4664; splits: train (3731), test (933)
- **EMBER** — total 800000; splits: train (600000), test (200000)
- **LRA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances out of the total number of instances. Reported as mean accuracy with standard deviation over 10-fold cross-validation for Kaggle and Drebin, and single-run accuracy for EMBER and LRA tasks.
- `execution_time` — range: seconds
  - Wall-clock time per training epoch or inference, measured in seconds. Models exceeding 10,000 seconds per epoch are marked as out-of-time (OOT).

## Input / output format

**Input**: Raw binary executables, assembly files, or APK/TAR archives converted into token sequences. Sequences are padded or truncated to a fixed maximum length (e.g., 256 to 131,072 tokens).

**Output**: Discrete class labels corresponding to malware families (for Kaggle/Drebin) or binary benign/malicious classification (for EMBER).

## Scoring recipe

```python
predictions = model.predict(sequences)
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
accuracy = correct / len(gold_labels)
return accuracy
```

## Common pitfalls

- Sequence length truncation/padding significantly impacts performance; models must handle variable-length binaries up to 100M+ tokens.
- Dataset splits are stratified by class (80-20 per class), not random global split, which affects cross-validation folds.
- LRA benchmark scores are poor proxies for real-world malware detection; high LRA accuracy does not guarantee strong malware classification performance.

## Evidence (verbatim from paper)

> To evaluate the performance, the proposed HGConv is compared with other state-of-the-art (SOTA) sequence models. For Kaggle and Drebin datasets, the proposed method is compared with non-attention-based processors... [Table 1] shows the mean accuracy with standard deviation for 10-fold cross-validation for each of the methods.

## Citation

```bibtex
@misc{alam2024holographic,
  title={Holographic Global Convolutional Networks for Long-Range Prediction Tasks in Malware Detection},
  author={Alam et al. (2024)},
  year={2024},
  note={arXiv:2403.17978}
}
```

- arXiv: 2403.17978