Tii Ssrc 23 Eval

---
name: tii-ssrc-23-eval
description: Evaluates the relative contribution of network traffic features to intrusion detection models by measuring performance degradation when features are shuffled. Probes the model's reliance on specific packet-level and flow-level statistics for distinguishing benign from malicious traffic and classifying attack types. Use when the user wants to benchmark on TII-SSRC-23, or asks about evaluating this task. Reports Permutation Feature Importance (PFI).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.10661
  bibtex_key: herzalla2023tiissrc23
  confidence: high
---

# tii-ssrc-23-eval

> TII-SSRC-23 Dataset: Typological Exploration of Diverse Traffic Patterns for Intrusion Detection — Herzalla et al. (2023) (arXiv:2310.10661, 2023)

## What this evaluates

Evaluates the relative contribution of network traffic features to intrusion detection models by measuring performance degradation when features are shuffled. Probes the model's reliance on specific packet-level and flow-level statistics for distinguishing benign from malicious traffic and classifying attack types.

## Datasets

- **TII-SSRC-23** — total ?; splits: full (-1)

## Metrics

- `Permutation Feature Importance (PFI)` **(primary)** — range: other
  - Measures the decrease in a model's performance score when a feature's values are randomly shuffled. A larger drop indicates higher feature importance. Evaluated across multiple classifiers and runs to ensure robustness.

## Input / output format

**Input**: CSV file containing 75 extracted features per bidirectional flow, with labels for 'Label' (Benign/Malicious), 'Traffic Type', and 'Traffic Subtype'.

**Output**: Ranked feature importance scores, typically aggregated and visualized as boxplots across multiple classifiers and runs.

## Scoring recipe

```python
def compute_pfi(model, X, y, score_func):
    base_score = score_func(model.predict(X), y)
    importances = []
    for col in X.columns:
        X_shuffled = X.copy()
        X_shuffled[col] = np.random.permutation(X_shuffled[col])
        shuffled_score = score_func(model.predict(X_shuffled), y)
        importances.append(base_score - shuffled_score)
    return importances
```

## Common pitfalls

- PFI scores can be inflated for correlated features, as shuffling one feature may not fully remove predictive power if correlated features remain.
- Importance rankings vary significantly across different classifier architectures, so relying on a single model yields biased results.
- Multiple random shuffles and runs are required to stabilize estimates, as a single execution can produce noisy importance scores.

## Evidence (verbatim from paper)

> We employed Permutation Feature Importance (PFI) to compute the feature importance. PFI works by randomly shuffling the values of one feature at a time and then evaluating the resultant effect on the model’s performance. A marked decrease in the model’s performance implies the shuffled feature’s importance for the predictive task in question. However, evaluating feature importance should not entirely depend on a singular execution of PFI. It is advisable to perform multiple runs per method and utilize various classifiers when assessing feature importance. ... We employed three classifiers to calculate feature importance: the Random Forest (RF) classifier, the eXtreme Gradient Boosting (XGBoost) classifier, and the Extra Trees (ET) classifier. ... for each classifier, we conducted three separate runs of PFI

## Citation

```bibtex
@misc{herzalla2023tiissrc23,
  title={TII-SSRC-23 Dataset: Typological Exploration of Diverse Traffic Patterns for Intrusion Detection},
  author={Herzalla et al. (2023)},
  year={2023},
  note={arXiv:2310.10661}
}
```

- arXiv: 2310.10661