Concap Nids Eval

---
name: concap-nids-eval
description: Evaluates the ability of machine learning and flow-based intrusion detection systems to accurately classify network traffic flows as benign or malicious. It probes the model's capacity to generalize across real-world benchmarks and synthetically generated, automatically labeled traffic for multi-step attack scenarios. Use when the user wants to benchmark on CICIDS17, ConCap ssh-patator, or asks about evaluating this task. Reports tpr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.16038
  bibtex_key: verkerken2025concap
  confidence: high
---

# concap-nids-eval

> ConCap: Practical Network Traffic Generation for (ML- and) Flow-based Intrusion Detection Systems — Verkerken et al. (2025) (arXiv:2509.16038, 2025)

## What this evaluates

Evaluates the ability of machine learning and flow-based intrusion detection systems to accurately classify network traffic flows as benign or malicious. It probes the model's capacity to generalize across real-world benchmarks and synthetically generated, automatically labeled traffic for multi-step attack scenarios.

## Datasets

- **CICIDS17** — total ?; splits: train (-1), test (-1)
- **ConCap ssh-patator** — total ?; splits: train (-1), test (-1); repo https://github.com/idlab-discover/ConCap

## Metrics

- `tpr` **(primary)** — range: [0, 1]
  - True Positive Rate (Recall) = TP / (TP + FN). Measures the proportion of actual malicious flows correctly identified by the detector.
- `fpr` — range: [0, 1]
  - False Positive Rate = FP / (FP + TN). Measures the proportion of benign flows incorrectly flagged as malicious.
- `accuracy` — range: [0, 1]
  - Accuracy = (TP + TN) / (TP + TN + FP + FN). Proportion of correctly classified flows out of the total test set.

## Input / output format

**Input**: CSV file containing NetFlow records with features such as Src Port, Dst Port, Protocol, Flow Duration, and Number of packets.

**Output**: Binary classification label per row: 0 for Benign, 1 for Malicious.

## Scoring recipe

```python
tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
tn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 0)
tpr = tp / (tp + fn) if (tp + fn) > 0 else 0.0
fpr = fp / (fp + tn) if (fp + tn) > 0 else 0.0
accuracy = (tp + tn) / len(gold) if len(gold) > 0 else 0.0
return {'tpr': tpr, 'fpr': fpr, 'accuracy': accuracy}
```

## Common pitfalls

- AutoEncoders require explicit thresholding on reconstruction error to output binary classifications; using a threshold derived from a different dataset (e.g., xNIDS) can cause unexpectedly high FPR on ConCap data.
- LLM zero-shot prompting without context fails to detect malicious SSH bursts, as repeated connections alone do not inherently imply malicious intent without domain-specific heuristics or augmented examples.
- The evaluation uses an 80/20 split on both benign and malicious subsets independently; researchers must ensure the held-out 20% test set is strictly disjoint from training to avoid data leakage.

## Evidence (verbatim from paper)

> Specifically, when trained on ConCap’s generated data (which is always malicious), the AE-IDS achieves $tpr=0.999$ on the testing partition of ConCap’s generated data, and $tpr=0.987$ on the testing partition of malicious data of the same attack included in CICIDS17; conversely, when trained on the malicious data of CICIDS17, the AE-IDS achieves $tpr=0.955$ on the testing partition of CICIDS17, and $tpr=0.952$ on the testing partition of ConCap’s generated data. The $fpr$ is 0.478 on the former case, and 0.209 in the latter case: such an underwhelming result is because our AE-IDS uses the same thresholding mechanism used in xNIDS (note that AutoEncoders are not classifiers: to use them in a classification task, one must specify a threshold on the reconstruction error. We used the one of xNIDS, which was derived on a different dataset. To improve the $fpr$, one can simply change the threshold).

## Citation

```bibtex
@misc{verkerken2025concap,
  title={ConCap: Practical Network Traffic Generation for (ML- and) Flow-based Intrusion Detection Systems},
  author={Verkerken et al. (2025)},
  year={2025},
  note={arXiv:2509.16038}
}
```

- arXiv: 2509.16038