Malicia Malware Classification Eval

---
name: malicia-malware-classification-eval
description: Evaluates a model's ability to classify malware samples into one of seven predefined families based on their opcode sequences. It probes the model's capacity to capture structural and sequential patterns in low-level code representations. Use when the user wants to benchmark on Malicia, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.11032
  bibtex_key: mehta2023malware
  confidence: high
---

# malicia-malware-classification-eval

> A Natural Language Processing Approach to Malware Classification — Mehta et al. (2023) (arXiv:2307.11032, 2023)

## What this evaluates

Evaluates a model's ability to classify malware samples into one of seven predefined families based on their opcode sequences. It probes the model's capacity to capture structural and sequential patterns in low-level code representations.

## Datasets

- **Malicia** — total 8054; splits: validation (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly classified samples out of the total number of samples.
- `weighted F1-score` — range: [0, 1]
  - Weighted average of per-class F1 scores, where weights correspond to the number of true instances for each class.

## Input / output format

**Input**: Opcode sequences extracted from malware samples, truncated to a fixed length L (25, 50, 100, or 200). Samples with fewer opcodes than L are dropped.

**Output**: A single class label from the seven malware families.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    classes = set(gold_labels)
    f1_scores, weights = [], []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
        weights.append(sum(1 for g in gold_labels if g == c))
    weighted_f1 = sum(f * w for f, w in zip(f1_scores, weights)) / sum(weights)
    return accuracy, weighted_f1
```

## Common pitfalls

- The dataset is highly imbalanced across the seven malware families, so overall accuracy can be misleading for minority classes.
- Samples with opcode sequences shorter than the chosen truncation length L are dropped, which may slightly alter the effective dataset size and distribution.
- The paper reports validation accuracy for hyperparameter tuning rather than a held-out test set, so results may reflect tuning bias.

## Evidence (verbatim from paper)

> The accuracy we obtained for the best choice of hyperparameters in Table 2 was 0.9758. Table 4 shows the accuracy and weighted F1-score obtained after testing the following techniques on the same seven families of the Malicia dataset.

## Citation

```bibtex
@misc{mehta2023malware,
  title={A Natural Language Processing Approach to Malware Classification},
  author={Mehta et al. (2023)},
  year={2023},
  note={arXiv:2307.11032}
}
```

- arXiv: 2307.11032