Droidspan Eval

---
name: droidspan-eval
description: Evaluates the sustainability and robustness of a dynamic behavioral profiling approach (DroidSpan) for Android malware detection over time and against code obfuscation, compared to a static baseline (MamaDroid). Use when the user wants to benchmark on all-data, oldBen+oldMal, MalObf, or asks about evaluating this task. Reports F1-measure.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1807.08221
  bibtex_key: cai2018preliminary
  confidence: high
---

# droidspan-eval

> A Preliminary Study On the Sustainability of Android Malware Detection — Cai (2018) (arXiv:1807.08221, 2018)

## What this evaluates

Evaluates the sustainability and robustness of a dynamic behavioral profiling approach (DroidSpan) for Android malware detection over time and against code obfuscation, compared to a static baseline (MamaDroid).

## Datasets

- **all-data** — total ?; splits: train (-1), test (-1)
- **oldBen+oldMal** — total ?; splits: train (-1)
- **MalObf** — total 220; splits: test (220)

## Metrics

- `Precision` — range: [0, 1]
  - P = TP / (TP + FP), computed with respect to the positive class L = MALICIOUS.
- `Recall` — range: [0, 1]
  - R = TP / (TP + FN), computed with respect to the positive class L = MALICIOUS.
- `F1-measure` **(primary)** — range: [0, 1]
  - F1 = 2 * (P * R) / (P + R). The paper notes that F1 scores in cross-validation are averaged across folds, not recomputed from averaged P and R.

## Input / output format

**Input**: Runtime dynamic traces of Android apps, specifically the Sensitive Access Distribution (SAD) profile capturing patterns of sensitive data and operation accesses.

**Output**: Binary classification label: BENIGN or MALICIOUS.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'MALICIOUS' and g == 'MALICIOUS')
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'MALICIOUS' and g != 'MALICIOUS')
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p != 'MALICIOUS' and g == 'MALICIOUS')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
```

## Common pitfalls

- F1 scores in cross-validation are averaged across folds, not recomputed from averaged precision and recall.
- MamaDroid's static analysis (FlowDroid) fails on obfuscated apps, making direct comparison on the MalObf dataset impossible.
- Evaluation splits are strictly time-based (training on older malware, testing on newer) to measure sustainability, not random splits.

## Evidence (verbatim from paper)

> In our evaluation, we gauged the performance of DroidSpan versus the baseline in terms of three metrics: precision, recall, and F1-measure (accuracy), as defined below. ... Precision (P) = TP/(TP + FP), Recall (R) = TP/(TP + FN), and F1 = 2 * (P*R)/(P+R). For evaluating malware detection, these metrics are computed concerning L = MALICIOUS only.

## Citation

```bibtex
@misc{cai2018preliminary,
  title={A Preliminary Study On the Sustainability of Android Malware Detection},
  author={Cai (2018)},
  year={2018},
  note={arXiv:1807.08221}
}
```

- arXiv: 1807.08221