Magic Apt Detection Eval

---
name: magic-apt-detection-eval
description: Evaluates the ability of a self-supervised graph representation learning model to detect Advanced Persistent Threats (APTs) in system audit logs. It probes multi-granularity anomaly detection (batched log-level and system entity-level) under a strict unsupervised setting where only benign data is available for training. Use when the user wants to benchmark on StreamSpot, Unicorn Wget, DARPA Engagement 3, or asks about evaluating this task. Reports Precision.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.09831
  bibtex_key: jia2023magic
  confidence: high
---

# magic-apt-detection-eval

> MAGIC: Detecting Advanced Persistent Threats via Masked Graph Representation Learning — Zian Jia et al. (2023) (arXiv:2310.09831, 2023)

## What this evaluates

Evaluates the ability of a self-supervised graph representation learning model to detect Advanced Persistent Threats (APTs) in system audit logs. It probes multi-granularity anomaly detection (batched log-level and system entity-level) under a strict unsupervised setting where only benign data is available for training.

## Datasets

- **StreamSpot** — total ?; splits: train (400), test (200)
- **Unicorn Wget** — total ?; splits: train (100), test (50)
- **DARPA Engagement 3** — total ?; splits: train (-1), test (-1)

## Metrics

- `Precision` **(primary)** — range: [0, 1]
  - Ratio of correctly predicted malicious instances to all instances predicted as malicious. Formula: TP / (TP + FP).
- `Recall` — range: [0, 1]
  - Ratio of correctly predicted malicious instances to all actual malicious instances. Formula: TP / (TP + FN).
- `FPR` — range: [0, 1]
  - False Positive Rate: Ratio of incorrectly predicted malicious instances to all actual benign instances. Formula: FP / (FP + TN).
- `F1-Score` — range: [0, 1]
  - Harmonic mean of Precision and Recall. Formula: 2 * (Precision * Recall) / (Precision + Recall).
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between classes across all thresholds.

## Input / output format

**Input**: System audit logs (system calls, file operations, network connections) aggregated into provenance graphs. Inputs are provided as batches of logs for log-level detection or individual system entities for entity-level detection.

**Output**: A detection score or binary classification label (benign/malicious) for each input log batch or system entity.

## Scoring recipe

```python
tp = sum(pred == 1 and gold == 1)
fp = sum(pred == 1 and gold == 0)
fn = sum(pred == 0 and gold == 1)
tn = sum(pred == 0 and gold == 0)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
auc = compute_auc(gold, scores) # from ROC curve
```

## Common pitfalls

- Training exclusively on benign data forces the model to learn normal behavior distributions rather than attack signatures, making it sensitive to concept drift and novel benign patterns.
- Highly imbalanced test sets (e.g., DARPA E3) can mask poor performance if accuracy is reported; FPR and Recall are the critical metrics here.
- Detection granularity varies significantly: batched log-level detection aggregates system calls, while entity-level detection isolates individual processes/files, leading to different performance characteristics (e.g., hard to detect passive files/libraries).

## Evidence (verbatim from paper)

> MAGIC yields low FPR (average 0.15%) with large training data. This is because MAGIC models benign system behaviors with self-supervised embeddings, allowing it to effectively handle unseen system entities.

## Citation

```bibtex
@misc{jia2023magic,
  title={MAGIC: Detecting Advanced Persistent Threats via Masked Graph Representation Learning},
  author={Zian Jia et al. (2023)},
  year={2023},
  note={arXiv:2310.09831}
}
```

- arXiv: 2310.09831