Mthl Network Traffic Eval

---
name: mthl-network-traffic-eval
description: Evaluates machine learning models on hierarchical network traffic classification tasks, including top-level protocol identification and malware detection, as well as mid-level application and malware type classification. Use when the user wants to benchmark on VPN-nonVPN + $(\mathsf{Net})^2$ + CICIDS2017, or asks about evaluating this task. Reports Macro-average F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.03850
  bibtex_key: barut2021mthl
  confidence: high
---

# mthl-network-traffic-eval

> Multi-Task Hierarchical Learning Based Network Traffic Analytics — Barut et al. (2021) (arXiv:2106.03850, 2021)

## What this evaluates

Evaluates machine learning models on hierarchical network traffic classification tasks, including top-level protocol identification and malware detection, as well as mid-level application and malware type classification.

## Datasets

- **VPN-nonVPN + $(\mathsf{Net})^2$ + CICIDS2017** — total ?; splits: train (-1), validation (-1); repo https://github.com/ACANETS/NetML-Competition2020

## Metrics

- `Macro-average F1 score` **(primary)** — range: [0, 1]
  - Computes the unweighted mean of the F1 score across all classes, providing a general insight for multi-class classification problems with imbalanced data.

## Input / output format

**Input**: Metadata features extracted from network flows (100+ engineered flow features).

**Output**: Hierarchical classification labels: top-level (protocol identification, malware detection) and mid-level (application classification, malware type classification).

## Scoring recipe

```python
def macro_f1(y_true, y_pred):
    classes = sorted(set(y_true) | set(y_pred))
    f1s = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)
```

## Common pitfalls

- The 20% split is explicitly labeled as a 'validation set' in the text but is used as the final evaluation set for baseline comparison.
- Only metadata features are used for all models, explicitly excluding protocol-specific features to expedite efficiency, which may limit performance compared to full-feature baselines.
- Macro-average F1 is chosen specifically to address class imbalance, meaning standard accuracy or micro-F1 would yield misleadingly high scores.

## Evidence (verbatim from paper)

> Macro-average F1 score is utilized to evaluate the performance since it provides more general insight for multi-class classification problems with imbalanced data. 80% of the training split is used to train the classifiers while the remaining 20% is reserved to for evaluation and referred as validation set.

## Citation

```bibtex
@misc{barut2021mthl,
  title={Multi-Task Hierarchical Learning Based Network Traffic Analytics},
  author={Barut et al. (2021)},
  year={2021},
  note={arXiv:2106.03850}
}
```

- arXiv: 2106.03850