Kairos Eval

---
name: kairos-eval
description: Evaluates a provenance-based intrusion detection system's ability to identify anomalous system behavior and reconstruct attack footprints from whole-system kernel-level logs. It probes the model's capacity to distinguish between benign and malicious activity in temporal windows without relying on attack signatures. Use when the user wants to benchmark on Manzoor et al., DARPA-E3-THEIA, DARPA-E3-CADETS, DARPA-E3-ClearScope, DARPA-E5-THEIA, DARPA-E5-CADETS, DARPA-E5-ClearScope, DARPA-OpTC, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.05034
  bibtex_key: cheng2023kairos
  confidence: high
---

# kairos-eval

> Kairos: Practical Intrusion Detection and Investigation using Whole-system Provenance — Cheng et al. (2023) (arXiv:2308.05034, 2023)

## What this evaluates

Evaluates a provenance-based intrusion detection system's ability to identify anomalous system behavior and reconstruct attack footprints from whole-system kernel-level logs. It probes the model's capacity to distinguish between benign and malicious activity in temporal windows without relying on attack signatures.

## Datasets

- **Manzoor et al.** — total 600; splits: train (5), val (120), test (475)
- **DARPA-E3-THEIA** — total ?; splits: train (-1), val (-1), test (-1)
- **DARPA-E3-CADETS** — total ?; splits: train (-1), val (-1), test (-1)
- **DARPA-E3-ClearScope** — total ?; splits: train (-1), val (-1), test (-1)
- **DARPA-E5-THEIA** — total ?; splits: train (-1), val (-1), test (-1)
- **DARPA-E5-CADETS** — total ?; splits: train (-1), val (-1), test (-1)
- **DARPA-E5-ClearScope** — total ?; splits: train (-1), val (-1), test (-1)
- **DARPA-OpTC** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Precision` — range: [0, 1]
  - Ratio of true positives to all predicted positives: TP / (TP + FP). Computed per 15-minute time window.
- `Recall` — range: [0, 1]
  - Ratio of true positives to all actual positives: TP / (TP + FN). Computed per 15-minute time window.
- `Accuracy` — range: [0, 1]
  - Ratio of correct predictions to total predictions: (TP + TN) / (TP + TN + FP + FN). Computed per 15-minute time window.
- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic (ROC) curve, measuring the model's ability to discriminate between benign and attack time windows across all classification thresholds.

## Input / output format

**Input**: Temporal kernel-level provenance graphs (nodes, edges, timestamps) aggregated into 15-minute time windows.

**Output**: Binary classification label per time window (benign or anomalous/attack), plus a reconstructed compact attack summary graph for investigation.

## Scoring recipe

```python
tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
tn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 0)
fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
accuracy = (tp + tn) / (tp + tn + fp + fn)
# AUC computed via ROC curve over time-window predictions
```

## Common pitfalls

- Ground truth often dismisses entities that remain active after an attack, causing the system to flag them as false positives even though they are still compromised.
- Metrics are computed per 15-minute time window, not per individual edge or node, which can mask fine-grained detection errors.
- Concept drift from new benign applications in test data triggers high reconstruction errors, requiring incremental retraining to maintain precision.

## Evidence (verbatim from paper)

> Table 4 shows the precision, recall, accuracy, and area under ROC curve (AUC) results for all datasets. We compute these metrics based on time windows. As mentioned in §5.1, we manually label each time window in a provenance graph as either benign or attack according to the ground truth. If KAIROS marks a benign time window as anomalous (i.e., if KAIROS mistakenly includes a benign time window in an anomalous queue), we consider the time window to be a false positive (FP). On the other hand, if KAIROS correctly marks an attack time window as anomalous, it is counted as a true positive (TP). False negatives (FN) and true negatives (TN) are calculated in a similar fashion.

## Citation

```bibtex
@misc{cheng2023kairos,
  title={Kairos: Practical Intrusion Detection and Investigation using Whole-system Provenance},
  author={Cheng et al. (2023)},
  year={2023},
  note={arXiv:2308.05034}
}
```

- arXiv: 2308.05034