Opcode Malware Classification Eval

---
name: opcode-malware-classification-eval
description: This evaluation probes the ability of machine learning and deep learning models to classify malware into specific families based on their assembly-level instruction sequences (opcodes). It compares traditional feature-engineering approaches using 1-gram and 2-gram n-grams against an end-to-end 1D CNN that processes raw opcode sequences. Use when the user wants to benchmark on OpCode Malware Dataset, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.13408
  bibtex_key: saini2025opcode
  confidence: medium
---

# opcode-malware-classification-eval

> OpCode-Based Malware Classification Using Machine Learning and Deep Learning Techniques — Saini et al. (2025) (arXiv:2504.13408, 2025)

## What this evaluates

This evaluation probes the ability of machine learning and deep learning models to classify malware into specific families based on their assembly-level instruction sequences (opcodes). It compares traditional feature-engineering approaches using 1-gram and 2-gram n-grams against an end-to-end 1D CNN that processes raw opcode sequences.

## Datasets

- **OpCode Malware Dataset** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly classified instances out of the total number of instances.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Traditional models: normalized 1-gram and 2-gram n-gram feature vectors derived from opcode sequences. CNN: raw opcode sequences.

**Output**: Predicted malware family label (categorical class).

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    accuracy = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    classes = set(gold)
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0)
    f1 = sum(f1_scores) / len(f1_scores)
    return {'accuracy': accuracy, 'f1_score': f1}
```

## Common pitfalls

- Class imbalance was explicitly mitigated using RandomOverSampler during training, so evaluation metrics should ideally be reported on the original imbalanced test set to reflect real-world deployment conditions.
- The protocol compares handcrafted n-gram features against raw sequence inputs, which can confound whether performance differences stem from the model architecture or the feature representation strategy.

## Evidence (verbatim from paper)

> SVM: Accuracy 66.37%, F1-score 64.04%  
- KNN: Accuracy 63.65%, F1-score 61.02%  
Decision Tree: Accuracy 62.14%, F1-score 60.06%  
- Voting Classifier: Accuracy:  $68.61\%$  (but did not outperform SVM).

# Deep Learning Approach:

The CNN is trained using PyTorch for 10 epochs with the Adam optimizer  $(\mathrm{lr} = 0.001)$  and a ReduceLROnPlateau scheduler. Evaluation metrics for the CNN are:

Accuracy:  $62.14\%$  
- Precision:  ${64.49}\%$  
- Recall:  $62.14\%$  
F1-score:  $60.44\%$

Comparative analysis indicates that while the CNN reduces the need for manual feature engineering, SVM remains the top-performing model on this dataset.

## Citation

```bibtex
@misc{saini2025opcode,
  title={OpCode-Based Malware Classification Using Machine Learning and Deep Learning Techniques},
  author={Saini et al. (2025)},
  year={2025},
  note={arXiv:2504.13408}
}
```

- arXiv: 2504.13408