Xai Feature Selection Ids Eval

---
name: xai-feature-selection-ids-eval
description: This evaluation probes the effectiveness of explainable AI (XAI)-driven feature selection methods on network intrusion detection systems. It measures how well various black-box machine learning models classify network traffic flows into normal or specific attack categories when trained on different subsets of extracted features. Use when the user wants to benchmark on CICIDS-2017, RoEduNet-SIMARGL2021, or asks about evaluating this task. Reports Accuracy (Acc).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.10050
  bibtex_key: arreche2024xai
  confidence: high
---

# xai-feature-selection-ids-eval

> XAI-based Feature Selection for Improved Network Intrusion Detection Systems — Arreche et al. (2024) (arXiv:2410.10050, 2024)

## What this evaluates

This evaluation probes the effectiveness of explainable AI (XAI)-driven feature selection methods on network intrusion detection systems. It measures how well various black-box machine learning models classify network traffic flows into normal or specific attack categories when trained on different subsets of extracted features.

## Datasets

- **CICIDS-2017** — total 2775364; splits: full (2775364)
- **RoEduNet-SIMARGL2021** — total 31433875; splits: full (31433875)

## Metrics

- `Accuracy (Acc)` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances out of the total number of instances.
- `F1-score (F1)` — range: [0, 1]
  - Harmonic mean of precision and recall, typically macro-averaged across classes.
- `Balanced Accuracy (Bacc)` — range: [0, 1]
  - Average of recall obtained on each class, accounting for class imbalance.
- `Matthews Correlation Coefficient (MCC)` — range: [-1, 1]
  - Correlation coefficient between observed and predicted binary classifications, robust to imbalance.
- `AUC-ROC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring trade-off between true positive and false positive rates.

## Input / output format

**Input**: Tabular network flow features including packet lengths, TCP flags, port numbers, flow durations, and inter-arrival times.

**Output**: Categorical label indicating traffic type: Normal, DoS, PortScan, Brute Force, Web Attack, Bot, or Infiltration.

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

def compute_metrics(y_true, y_pred, y_prob=None):
    acc = accuracy_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred, average='macro')
    auc = roc_auc_score(y_true, y_prob, multi_class='ovr') if y_prob is not None else None
    return {'Acc': acc, 'F1': f1, 'AUC-ROC': auc}
```

## Common pitfalls

- The datasets are severely imbalanced (e.g., CICIDS-2017 is 84.4% Normal traffic), making Accuracy alone misleading without macro-averaged F1 or Balanced Accuracy.
- The paper uses a custom weighted ranking system (3/2/1 points for top-3 methods across k=5, 10, 15 feature subsets) to compare feature selection techniques, which is non-standard and lacks a publicly available implementation.
- No explicit train/validation/test split is defined in the text, preventing exact reproduction of the reported performance numbers.

## Evidence (verbatim from paper)

> Black-box AI Metrics: We use a commonly used set of criteria for assessing intrusion detection and classification issues to evaluate the performance of the black-box AI models. The measures that fall under this category include balanced accuracy (Bacc), Matthews correlation coefficient (MCC), accuracy (Acc), precision (Prec), recall (Rec), and F1-score (F1). The effectiveness of the AI model is also determined by calculating the AucRoc (area under the ROC curve) score.

## Citation

```bibtex
@misc{arreche2024xai,
  title={XAI-based Feature Selection for Improved Network Intrusion Detection Systems},
  author={Arreche et al. (2024)},
  year={2024},
  note={arXiv:2410.10050}
}
```

- arXiv: 2410.10050