Ids Ensemble Eval

---
name: ids-ensemble-eval
description: Evaluates the ability of individual machine learning classifiers and ensemble strategies to detect network intrusions and classify traffic types. It probes model robustness, precision-recall trade-offs, and computational efficiency across diverse real-world network traffic datasets with varying attack profiles. Use when the user wants to benchmark on RoEduNet-SIMARGL2021, CICIDS-2017, or asks about evaluating this task. Reports F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.15597
  bibtex_key: [REDACTED]
  confidence: high
---

# ids-ensemble-eval

> A Comprehensive Comparative Study of Individual ML Models and Ensemble Strategies for Network Intrusion Detection Systems — Bibers et al. (2024) (arXiv:2410.15597, 2024)

## What this evaluates

Evaluates the ability of individual machine learning classifiers and ensemble strategies to detect network intrusions and classify traffic types. It probes model robustness, precision-recall trade-offs, and computational efficiency across diverse real-world network traffic datasets with varying attack profiles.

## Datasets

- **RoEduNet-SIMARGL2021** — total 30000000; splits: full (30000000)
- **CICIDS-2017** — total ?; splits: full (-1)

## Metrics

- `F1 Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * TP / (2 * TP + FP + FN). (Note: The paper contains a typographical error in the denominator, listing 2TN instead of 2TP).
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified instances: (TP + TN) / Total.
- `Precision` — range: [0, 1]
  - Proportion of true positives among all positive predictions: TP / (FP + TP).
- `Recall` — range: [0, 1]
  - Proportion of true positives among all actual positives: TP / (FN + TP).
- `Runtime` — range: other
  - Wall-clock time in seconds required for both model training and inference/testing phases.

## Input / output format

**Input**: Tabular network flow features derived from real-time traffic analysis (Netflow-like schema), including numerical and categorical attributes representing packet statistics, protocol types, and connection durations.

**Output**: Discrete class label indicating traffic type (e.g., Normal, Denial of Service, Malware, Port Scanning, Web Attack, etc.).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p and t == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == p and t == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * tp / (2 * tp + fp + fn) if (2 * tp + fp + fn) > 0 else 0
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1}
```

## Common pitfalls

- The paper defines F1-Score with a typo in the denominator (`2TN` instead of `2TP`), which would yield incorrect values if implemented literally.
- Runtime measurements combine training and testing phases, unlike standard benchmarks that report inference-only latency.
- Several ensemble models (Blending, Stacking, XGBoost, GB) were evaluated on only 20% of the RoEduNet-SIMARGL2021 dataset due to memory constraints, potentially skewing performance comparisons.

## Evidence (verbatim from paper)

> In this study, the utilization of well-established evaluation metrics is crucial to ascertain the most effective model for integration within an Intrusion Detection System (IDS). Accuracy, precision, recall, and F1-score stand as quintessential performance evaluation metrics. These metrics are derived from four fundamental measures: true positive (TP), false positive (FP), true negative (TN), and false negative (FN) rates. The evaluation metrics are delineated as follows: Accuracy [(TP+TN)/Total]: Signifies the proportion of accurately identified network traffic instances over the total data instances. Precision [TP/(FP+TP)]: Measures the frequency with which the model accurately discerns an attack. Recall [TP/(FN+TP)]: Measures the model’s ability to correctly identify attacks (or intrusions). F1-Score [2TP/(2TN+FP+FN)]: Represents the harmonic mean of precision and recall.

## Citation

```bibtex
@misc{bibers2024comprehensive,
  title={A Comprehensive Comparative Study of Individual ML Models and Ensemble Strategies for Network Intrusion Detection Systems},
  author={Bibers et al. (2024)},
  year={2024},
  note={arXiv:2410.15597}
}
```

- arXiv: 2410.15597