Cicids2017 Adversarial Eval

---
name: cicids2017-adversarial-eval
description: Evaluates the adversarial robustness of tree ensemble models (RF, XGB, LGBM, EBM) on enterprise network intrusion detection using the CICIDS2017 dataset. It measures how well models maintain detection performance on benign and malicious traffic when subjected to constrained adversarial perturbations of time-series traffic features. Use when the user wants to benchmark on CICIDS2017, or asks about evaluating this task. Reports F1S.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.16912
  bibtex_key: vitorino2024adversarial
  confidence: high
---

# cicids2017-adversarial-eval

> An Adversarial Robustness Benchmark for Enterprise Network Intrusion Detection — Vitorino et al. (2024) (arXiv:2402.16912, 2024)

## What this evaluates

Evaluates the adversarial robustness of tree ensemble models (RF, XGB, LGBM, EBM) on enterprise network intrusion detection using the CICIDS2017 dataset. It measures how well models maintain detection performance on benign and malicious traffic when subjected to constrained adversarial perturbations of time-series traffic features.

## Datasets

- **CICIDS2017** — total ?; splits: test (-1)

## Metrics

- `F1S` **(primary)** — range: percent
  - Harmonic mean of precision and recall. Calculated as 2 * (precision * recall) / (precision + recall). Reported in percent.
- `ACC` — range: percent
  - Accuracy, the proportion of correctly classified samples (both benign and malicious). Reported in percent.
- `PRC` — range: percent
  - Precision, the proportion of predicted malicious flows that are actually malicious. Reported in percent.
- `RCL` — range: percent
  - Recall, the proportion of actual malicious flows correctly identified. Reported in percent.
- `FPR` — range: percent
  - False Positive Rate, the proportion of benign flows incorrectly classified as malicious. Reported in percent.

## Input / output format

**Input**: Network traffic flows represented by 24 time-related features.

**Output**: Binary classification label: malicious or benign.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    prc = tp / (tp + fp) * 100 if (tp + fp) > 0 else 0
    rcl = tp / (tp + fn) * 100 if (tp + fn) > 0 else 0
    f1s = 2 * prc * rcl / (prc + rcl) if (prc + rcl) > 0 else 0
    acc = (tp + tn) / (tp + tn + fp + fn) * 100
    fpr = fp / (fp + tn) * 100 if (fp + tn) > 0 else 0
    return {'ACC': acc, 'PRC': prc, 'RCL': rcl, 'F1S': f1s, 'FPR': fpr}
```

## Common pitfalls

- Adversarial perturbations targeting time-series features cause near-total collapse of precision/recall for regularly trained models.
- Adversarial training improves robustness on the training distribution but may fail to generalize to newer, more complex attacks (e.g., HIKARI dataset).
- False Positive Rate (FPR) must be kept extremely low (<0.4%) to avoid costly enterprise false alarms, which is a key evaluation criterion alongside F1S.

## Evidence (verbatim from paper)

> The evaluation considers the regular holdout set of the CICIDS2017, NewCICIDS, and HIKARI datasets, as well as the model-specific adversarial holdout sets. ... The ACC, PRC, RCL, F1S, and FPR columns correspond to accuracy, precision, recall, F1-score, and false positive rate.

## Citation

```bibtex
@misc{vitorino2024adversarial,
  title={An Adversarial Robustness Benchmark for Enterprise Network Intrusion Detection},
  author={Vitorino et al. (2024)},
  year={2024},
  note={arXiv:2402.16912}
}
```

- arXiv: 2402.16912