Webapp Implementing Ebpf Security Monitoring

---
name: webapp-implementing-ebpf-security-monitoring
description: Implements eBPF-based security monitoring using Cilium Tetragon for real-time process execution tracking, network connection observability, file access auditing, and runtime enforcement.
domain: webapp
category: security-operations
hide: true
license: Apache-2.0
metadata:
  manifest_entry: src/corpus/webapp-implementing-ebpf-security-monitoring/SKILL.md
---
# Implementing eBPF Security Monitoring

## When to Use

- When deploying kernel-level runtime security monitoring on Linux hosts or Kubernetes clusters
- When you need sub-millisecond visibility into process execution, network connections, and file access
- When traditional userspace monitoring tools introduce unacceptable performance overhead
- When building detection pipelines that require in-kernel filtering before events reach userspace
- When enforcing runtime security policies (kill process, send signal) at the kernel level

## Common Misconfigurations & Verification

- **No BTF = silent no-load:** Tetragon needs kernel BTF (`/sys/kernel/btf/vmlinux`) or a matching external BTF for kprobe arg typing. On older/stripped kernels (no `CONFIG_DEBUG_INFO_BTF`) probes fail to attach and you get zero kprobe events while `process_exec` still flows — masking the gap. Verify `ls /sys/kernel/btf/vmlinux` and check `kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon` for attach errors.
- **Ring-buffer/event drops under load:** high event volume overflows the perf/ring buffer and Tetragon drops events — a coverage hole that looks like "nothing happened." Watch `tetragon_*_events_total` and missed/`map_errors` metrics, scope policies with in-kernel `matchArgs`/`matchBinaries` selectors (filter in-kernel, not in userspace), and avoid an unfiltered `tcp_connect` Post on a busy host.
- **Probe coverage gaps:** a syscall-name hook (`sys_execve`) can be bypassed by a different entry point or arch (compat/32-bit, `execveat`); matching on a fragile `Postfix`/`Prefix` string (e.g., `xmrig`) is trivially evaded by renaming the binary. Prefer hashes/paths and hook stable internal funcs (`commit_creds`, `fd_install`) over syscall names where possible.
- **Namespaced policy scoping mistakes:** a `TracingPolicyNamespaced` only applies to its namespace — host-level or other-namespace activity goes unmonitored if you assumed cluster-wide.
- **Enforcement risk:** a `Sigkill` action on a too-broad selector can kill legitimate processes — stage every enforcement policy in `Post` (observe) mode first and review events before switching to `Sigkill`.
- **Verification:** apply a policy, then deliberately trip it (e.g., `cat /etc/shadow`, `curl` to an external IP, exec a renamed `xmrig`) and confirm the matching `process_kprobe` event appears in `tetra getevents -o json` and lands in your SIEM sink before trusting the control.

## Prerequisites

- Linux kernel 5.3+ with BTF (BPF Type Format) support enabled
- Kubernetes 1.24+ cluster (for Kubernetes deployment) or standalone Linux host
- Helm 3.x installed (for Kubernetes deployment)
- `kubectl` configured with cluster access
- `tetra` CLI installed for local event streaming
- Python 3.8+ with `requests`, `kubernetes`, `pyyaml` dependencies
- Root or CAP_BPF/CAP_SYS_ADMIN capabilities for eBPF program loading

## Instructions

### 1. Install Tetragon on Kubernetes

Deploy Tetragon via Helm to get default process lifecycle observability:

```bash
helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system \
  --set tetragon.enableProcessCred=true \
  --set tetragon.enableProcessNs=true
```

Verify the installation:

```bash
kubectl get pods -n kube-system -l app.kubernetes.io/name=tetragon
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | head -20
```

### 2. Install Tetragon on Standalone Linux

For non-Kubernetes Linux hosts, install from the tarball release:

```bash
curl -LO https://github.com/cilium/tetragon/releases/latest/download/tetragon-linux-amd64.tar.gz
tar xzf tetragon-linux-amd64.tar.gz
sudo cp tetragon /usr/local/bin/
sudo cp tetra /usr/local/bin/

# Start tetragon daemon
sudo tetragon --btf /sys/kernel/btf/vmlinux &

# Stream events
tetra getevents -o compact
```

### 3. Monitor Process Execution (Default)

Tetragon generates `process_exec` and `process_exit` events by default without any TracingPolicy:

```bash
# Stream process events in compact format
tetra getevents -o compact

# Stream in JSON for SIEM ingestion
tetra getevents -o json | jq '.process_exec // .process_exit'
```

Example `process_exec` JSON event:

```json
{
  "process_exec": {
    "process": {
      "binary": "/usr/bin/curl",
      "arguments": "https://malicious.example.com/payload",
      "cwd": "/tmp",
      "uid": 1000,
      "pod": {
        "namespace": "default",
        "name": "webapp-7b4d9f8c6-x2k9p"
      },
      "parent": {
        "binary": "/bin/bash",
        "pid": 1234
      }
    }
  }
}
```

### 4. Author TracingPolicy for File Access Monitoring

Create a TracingPolicy CRD to monitor access to sensitive files via the `sys_openat` kprobe:

```yaml
# file-access-monitor.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: monitor-sensitive-file-access
spec:
  kprobes:
    - call: "fd_install"
      syscall: false
      args:
        - index: 0
          type: "int"
        - index: 1
          type: "file"
      selectors:
        - matchArgs:
            - index: 1
              operator: "Prefix"
              values:
                - "/etc/shadow"
                - "/etc/passwd"
                - "/etc/sudoers"
                - "/root/.ssh/"
                - "/etc/kubernetes/pki/"
          matchActions:
            - action: Post
```

Apply and observe:

```bash
kubectl apply -f file-access-monitor.yaml
tetra getevents -o compact --process-filter "event_set:PROCESS_KPROBE"
```

### 5. Author TracingPolicy for Network Connection Monitoring

Monitor outbound TCP connections using the `tcp_connect` kprobe:

```yaml
# network-monitor.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: monitor-tcp-connections
spec:
  kprobes:
    - call: "tcp_connect"
      syscall: false
      args:
        - index: 0
          type: "sock"
      selectors:
        - matchActions:
            - action: Post
```

### 6. Author TracingPolicy for Privilege Escalation Detection

Detect setuid/setgid calls that may indicate privilege escalation:

```yaml
# privilege-escalation-detect.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-privilege-escalation
spec:
  kprobes:
    - call: "__sys_setuid"
      syscall: false
      args:
        - index: 0
          type: "int"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Equal"
              values:
                - "0"
          matchActions:
            - action: Post
    - call: "commit_creds"
      syscall: false
      args:
        - index: 0
          type: "cred"
      selectors:
        - matchActions:
            - action: Post
```

### 7. Runtime Enforcement with Sigkill Action

Block unauthorized binary execution by killing the process in-kernel:

```yaml
# enforce-binary-allowlist.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: enforce-no-crypto-miners
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Postfix"
              values:
                - "xmrig"
                - "minerd"
                - "cpuminer"
                - "cryptonight"
          matchActions:
            - action: Sigkill
```

### 8. Export Events to SIEM

Configure Tetragon to export JSON events to a file sink for Fluentd/Filebeat/Vector ingestion:

```bash
# Helm values for file export
helm upgrade tetragon cilium/tetragon -n kube-system \
  --set tetragon.exportFilename=/var/log/tetragon/tetragon.log \
  --set tetragon.exportFileMaxSizeMB=100 \
  --set tetragon.exportFileMaxBackups=5
```

Then configure your log shipper (e.g., Filebeat) to tail `/var/log/tetragon/tetragon.log` and send to your SIEM.

### 9. Kubernetes-Aware Namespace Filtering

Use `TracingPolicyNamespaced` to scope monitoring to specific namespaces:

```yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
  name: monitor-production-file-access
  namespace: production
spec:
  kprobes:
    - call: "fd_install"
      syscall: false
      args:
        - index: 0
          type: "int"
        - index: 1
          type: "file"
      selectors:
        - matchArgs:
            - index: 1
              operator: "Prefix"
              values:
                - "/etc/shadow"
                - "/etc/passwd"
```

## Examples

### Detect Reverse Shell Connections

```yaml
# reverse-shell-detect.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-reverse-shells
spec:
  kprobes:
    - call: "tcp_connect"
      syscall: false
      args:
        - index: 0
          type: "sock"
      selectors:
        - matchBinaries:
            - operator: "In"
              values:
                - "/bin/bash"
                - "/bin/sh"
                - "/usr/bin/python3"
                - "/usr/bin/perl"
                - "/usr/bin/nc"
                - "/usr/bin/ncat"
          matchActions:
            - action: Post
```

### Monitor Container Escape Attempts

```yaml
# container-escape-detect.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-container-escape
spec:
  kprobes:
    - call: "sys_openat"
      syscall: true
      args:
        - index: 0
          type: "int"
        - index: 1
          type: "string"
      selectors:
        - matchArgs:
            - index: 1
              operator: "Prefix"
              values:
                - "/proc/1/root"
                - "/proc/1/ns"
                - "/sys/kernel/security"
                - "/proc/sysrq-trigger"
          matchActions:
            - action: Post
    - call: "sys_mount"
      syscall: true
      args:
        - index: 0
          type: "string"
        - index: 1
          type: "string"
        - index: 2
          type: "string"
      selectors:
        - matchActions:
            - action: Post
```

### Full Event Pipeline: Tetragon to Elasticsearch

```bash
# Use tetra CLI to pipe events through jq into Elasticsearch
tetra getevents -o json | jq -c 'select(.process_kprobe != null)' | \
  while IFS= read -r line; do
    curl -s -X POST "http://elasticsearch:9200/tetragon-events/_doc" \
      -H "Content-Type: application/json" \
      -d "$line"
  done
```