Ids Moo Automl Eval

---
name: ids-moo-automl-eval
description: Evaluates intrusion detection systems for resource-constrained IoT and cloud environments by measuring classification accuracy, computational efficiency, and model confidence. It probes the ability of AutoML pipelines to balance detection performance against training time, inference latency, and memory footprint. Use when the user wants to benchmark on CICIDS2017, IoTID20, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.08491
  bibtex_key: yang2025toward
  confidence: high
---

# ids-moo-automl-eval

> Toward Autonomous and Efficient Cybersecurity: A Multi-Objective AutoML-based Intrusion Detection System — Li Yang et al. (2025) (arXiv:2511.08491, 2025)

## What this evaluates

Evaluates intrusion detection systems for resource-constrained IoT and cloud environments by measuring classification accuracy, computational efficiency, and model confidence. It probes the ability of AutoML pipelines to balance detection performance against training time, inference latency, and memory footprint.

## Datasets

- **CICIDS2017** — total 26800; splits: train (-1), test (-1)
- **IoTID20** — total 31289; splits: train (-1), test (-1)

## Metrics

- `accuracy` — range: percent
  - Percentage of correctly classified samples out of the total test set.
- `precision` — range: percent
  - Ratio of true positive predictions to all positive predictions (weighted average across classes).
- `recall` — range: percent
  - Ratio of true positive predictions to all actual positives (weighted average across classes).
- `F1-score` **(primary)** — range: percent
  - Harmonic mean of precision and recall; explicitly cited as the comprehensive classification metric for evaluating detection effectiveness.
- `training time` — range: other
  - Total wall-clock time required to train the model on the training set, measured in seconds.
- `inference time` — range: other
  - Average time taken to classify a single network traffic sample on the test set, measured in milliseconds.
- `model size` — range: other
  - Disk footprint of the serialized model file, measured in megabytes.
- `average prediction probability` — range: percent
  - Mean probability assigned to the true class across all test samples, indicating model reliability.
- `Expected Calibration Error (ECE)` — range: percent
  - Quantifies the gap between predicted confidence and actual accuracy across confidence bins; lower values indicate better calibration.

## Input / output format

**Input**: Fixed-length feature vectors extracted from network traffic flows (e.g., 83 features for IoTID20), representing benign or malicious activity.

**Output**: Predicted class label (e.g., Normal, DoS, Web-Attack) and associated confidence/probability scores.

## Scoring recipe

```python
def evaluate(y_true, y_pred, y_prob):
    acc = (y_true == y_pred).mean()
    prec, rec, f1 = precision_recall_fscore_support(y_true, y_pred, average='weighted')
    avg_conf = np.mean(np.max(y_prob, axis=1))
    bins = np.linspace(0, 1, 10)
    ece = 0.0
    for i in range(len(bins)-1):
        mask = (y_prob.max(axis=1) >= bins[i]) & (y_prob.max(axis=1) < bins[i+1])
        if mask.sum() > 0:
            ece += mask.sum() * abs(y_true[mask].mean() - y_prob[mask].max(axis=1).mean())
    ece /= len(y_true)
    return {'accuracy': acc, 'precision': prec, 'recall': rec, 'f1': f1,
            'avg_confidence': avg_conf, 'ece': ece}
```

## Common pitfalls

- Hardware dependency: Execution times and model sizes are measured on a specific Dell Precision 3630 machine and do not generalize across different CPU/RAM configurations.
- Class imbalance: Datasets contain highly skewed attack distributions; accuracy alone can be misleading, necessitating stratified sampling and weighted F1-score evaluation.
- Confidence calibration: ECE and average prediction probability require well-calibrated probability outputs; models optimized solely for accuracy may yield poorly calibrated confidence scores.

## Evidence (verbatim from paper)

> For comprehensively evaluating the proposed model’s effectiveness, four performance measures are used, including accuracy, precision, recall, and F1-scores, as network traffic data are often highly imbalanced and contain only a small proportion of attack samples [[24]]. Additionally, since the proposed IDS emphasizes the balance between model effectiveness and efficiency, the execution time of the proposed model, including the total training time on the training set and the inference time per sample on the test set, is utilized to evaluate the proposed model’s efficiency.

## Citation

```bibtex
@misc{yang2025toward,
  title={Toward Autonomous and Efficient Cybersecurity: A Multi-Objective AutoML-based Intrusion Detection System},
  author={Li Yang et al. (2025)},
  year={2025},
  note={arXiv:2511.08491}
}
```

- arXiv: 2511.08491