Ids Automl Eval

---
name: ids-automl-eval
description: Evaluates an AutoML-based intrusion detection system's ability to classify network traffic as benign or malicious across multiple attack types. It probes the framework's robustness to class imbalance and its efficiency in real-time network environments. Use when the user wants to benchmark on CICIDS2017, 5G-NIDD, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.03141
  bibtex_key: yang2024towards
  confidence: high
---

# ids-automl-eval

> Towards Autonomous Cybersecurity: An Intelligent AutoML Framework for Autonomous Intrusion Detection — Li Yang et al. (2024) (arXiv:2409.03141, 2024)

## What this evaluates

Evaluates an AutoML-based intrusion detection system's ability to classify network traffic as benign or malicious across multiple attack types. It probes the framework's robustness to class imbalance and its efficiency in real-time network environments.

## Datasets

- **CICIDS2017** — total ?; splits: test (-1)
- **5G-NIDD** — total ?; splits: test (-1)

## Metrics

- `accuracy` — range: percent
  - Proportion of correctly classified samples: (TP + TN) / (TP + TN + FP + FN).
- `precision` — range: percent
  - Proportion of positive predictions that are correct: TP / (TP + FP).
- `recall` — range: percent
  - Proportion of actual positives correctly identified: TP / (TP + FN).
- `F1-score` **(primary)** — range: percent
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
- `training time` — range: other
  - Wall-clock time required to train the final ensemble model.
- `average test time per sample` — range: other
  - Total inference time divided by the number of test samples.

## Input / output format

**Input**: Tabular network traffic features extracted from captured packets.

**Output**: Classification label indicating whether traffic is benign or a specific attack type (e.g., DoS, botnet, brute force, infiltration, port scan, web attacks).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return accuracy, precision, recall, f1
```

## Common pitfalls

- Datasets are relatively simple, allowing many baselines to exceed 99% accuracy, which may overstate real-world gains.
- Class imbalance is addressed via synthetic data balancing (TVAE) before training, but metrics are reported on the original test distribution.
- Training time is reported but inference time per sample is emphasized as more critical for edge deployment.

## Evidence (verbatim from paper)

> To evaluate the proposed AutoML-based IDS framework, two public benchmark network traffic datasets, namely CICIDS2017 and 5G-NIDD, are utilized in the experiments. ... Due to the inherent class imbalance issues in network intrusion detection datasets, four model performance metrics—accuracy, precision, recall, and F1-scores—are considered collectively in the experiments. The F1-score is utilized as the primary performance metric in the performance-based automated model selection and tuning process of the proposed AutoML framework, as it offers a balanced view of anomaly detection results by calculating the harmonic mean of recall and precision. Additionally, the model execution time, involving the training and inference time of the final OCSE model, is utilized to assess the model’s efficiency.

## Citation

```bibtex
@misc{yang2024towards,
  title={Towards Autonomous Cybersecurity: An Intelligent AutoML Framework for Autonomous Intrusion Detection},
  author={Li Yang et al. (2024)},
  year={2024},
  note={arXiv:2409.03141}
}
```

- arXiv: 2409.03141