K8s Misconfig Detection Eval

---
name: k8s-misconfig-detection-eval
description: Evaluates the ability of security tools to detect network misconfigurations in Kubernetes Helm charts, focusing on discrepancies between declared configurations and actual runtime behavior, including port exposure, label collisions, and network policy effectiveness. Use when the user wants to benchmark on Kubernetes Helm Charts (287 apps), or asks about evaluating this task. Reports misconfiguration detection (found/partially found/missed).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.21134
  bibtex_key: bufalino2025insidejob
  confidence: high
---

# k8s-misconfig-detection-eval

> Inside Job: Defending Kubernetes Clusters Against Network Misconfigurations — Bufalino et al. (2025) (arXiv:2506.21134, 2025)

## What this evaluates

Evaluates the ability of security tools to detect network misconfigurations in Kubernetes Helm charts, focusing on discrepancies between declared configurations and actual runtime behavior, including port exposure, label collisions, and network policy effectiveness.

## Datasets

- **Kubernetes Helm Charts (287 apps)** — total 287; splits: analysis (287)

## Metrics

- `misconfiguration detection (found/partially found/missed)` **(primary)** — range: other
  - Counts the number of applications exhibiting each of the 15 defined misconfiguration types (M1–M7). For tool comparison, outputs are categorized as ● found, ○ partially found, × missed, or - not applicable per misconfiguration type.

## Input / output format

**Input**: Helm chart YAML files (static) and deployed Kubernetes clusters (runtime). Applications are deployed individually in an isolated Minikube environment (Kubernetes 1.25) to observe runtime behavior.

**Output**: A list of detected misconfigurations categorized into 15 types (M1–M7 subtypes), indicating whether each misconfiguration is present, partially present, or absent in the target application.

## Scoring recipe

```python
# Ground truth establishment
static_issues = parse_helm_yaml(chart)
runtime_issues = observe_minikube_cluster(chart)
ground_truth = merge(static_issues, runtime_issues)

# Tool evaluation
for misconfig_type in M1..M7:
    if tool_output.contains(misconfig_type):
        score = "found"
    elif tool_output.partially_contains(misconfig_type):
        score = "partially found"
    else:
        score = "missed"
return score_matrix
```

## Common pitfalls

- Dynamic ports (M2) change on every restart, requiring at least two runtime analysis iterations to detect reliably.
- Host network access (M7) exposes all host ports, necessitating a baseline port scan to subtract unrelated services before reporting.
- Static-only tools cannot detect runtime-only misconfigurations, while runtime-only tools miss cluster-wide label/selector collisions.

## Evidence (verbatim from paper)

> We take a Helm chart as input then carry out both static and runtime analysis. We then combine the obtained results and evaluate them against the machine-readable rules. ... The symbols indicate whether these were ● found, ○ partially found (i.e., either find less misconfigurations or require multiple runs), × missed, or - not applicable (i.e., could not be found by intrinsic limitations in the type of tool).

## Citation

```bibtex
@misc{bufalino2025insidejob,
  title={Inside Job: Defending Kubernetes Clusters Against Network Misconfigurations},
  author={Bufalino et al. (2025)},
  year={2025},
  note={arXiv:2506.21134}
}
```

- arXiv: 2506.21134