Big2015 Eval

---
name: big2015-eval
description: Evaluates the ability of deep learning models to classify malware binaries into their respective family types using image-based representations, specifically probing performance on imbalanced class distributions. Use when the user wants to benchmark on BIG2015, or asks about evaluating this task. Reports F-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1912.10836
  bibtex_key: cayir2019randomcapsnet
  confidence: high
---

# big2015-eval

> Random CapsNet Forest Model for Imbalanced Malware Type Classification Task — Çayır et al. (2019) (arXiv:1912.10836, 2019)

## What this evaluates

Evaluates the ability of deep learning models to classify malware binaries into their respective family types using image-based representations, specifically probing performance on imbalanced class distributions.

## Datasets

- **BIG2015** — total 10868; splits: train (8151), val (1359), test (1358)

## Metrics

- `accuracy` — range: [0, 1]
  - Ratio of correct predictions to total instances: (TP+TN)/(TP+TN+FP+FN).
- `F-Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: (2×TP)/(2×TP+FN+FP). The paper provides the binary formula but applies it to multi-class malware families; averaging strategy is not explicitly specified.

## Input / output format

**Input**: Image representation of malware binaries.

**Output**: Predicted malware family class label.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == g)
    accuracy = tp / len(gold)
    # F-Score formula given is binary; multi-class averaging unspecified
    # Standard implementation for multi-class:
    f1 = f1_score(gold, preds, average='macro')
    return accuracy, f1
```

## Common pitfalls

- The F-Score formula provided is strictly for binary classification, but the task is multi-class; the paper does not specify whether macro, micro, or weighted averaging is used.
- Stratified splits are mentioned, but random seeds and exact shuffling procedures are not reported, making exact replication of train/val/test partitions difficult.

## Evidence (verbatim from paper)

> Model evaluation has been done in terms of accuracy, F-Score, and the number of parameters of deep neural nets. These performance metrics are defined as follows: accuracy=(TP+TN)/(TP+TN+FP+FN) ... F-Score=(2×TP)/(2×TP+FN+FP) where true positive (TP) and false positive (FP) are the numbers of instances correctly and wrongly classified as positive respectively.

## Citation

```bibtex
@misc{cayir2019randomcapsnet,
  title={Random CapsNet Forest Model for Imbalanced Malware Type Classification Task},
  author={Çayır et al. (2019)},
  year={2019},
  note={arXiv:1912.10836}
}
```

- arXiv: 1912.10836