Splunk Connect For Syslog Setup

---
name: splunk-connect-for-syslog-setup
description: "Use when the user asks about SC4S, Splunk Connect for Syslog, syslog-ng collector setup, or syslog
  ingestion through HEC. Deploy and validate Splunk Connect for Syslog (SC4S) for Splunk Enterprise or
  Splunk Cloud. Prepares Splunk indexes and HEC, renders Docker/Podman/systemd or Kubernetes Helm
  configuration, and validates SC4S startup."
compatibility: "Splunk Cloud Platform 10.5.2605: conditional. Follow documented package, entitlement, topology, and customer-managed runtime guardrails; self-managed paths remain on the public 10.4 baseline."
metadata:
  splunk_cloud_10_5: "conditional"
  compatibility_verified: "2026-08-20"
---

# Splunk Connect for Syslog Setup

## Prerequisites

| Tool or access | Purpose | Verify |
|---|---|---|
| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |
| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |
| Credential files for live modes | Keep secrets out of chat | Verify paths only |

## Workflow Overview

```text
┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘
```

## When to Activate

- SC4S, Splunk Connect for Syslog, syslog-ng collector setup, or syslog ingestion through HEC.
- Preview and review the splunk connect for syslog setup workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

## Scope

Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.

## Examples

Inspect the supported setup modes before selecting one:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh --help
```

Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.

Inspect validation modes before running completion checks:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/validate.sh --help
```

Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.

## Troubleshooting

| Issue | Cause | Resolution |
|---|---|---|
| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |
| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |
| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |
| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |

Automates the operator workflow for **Splunk Connect for Syslog** (`SC4S`), an
external syslog-ng based collector that forwards events to Splunk over HEC.

## How SC4S Fits This Repo

SC4S is not a normal Splunkbase TA install. The skill handles two separate
areas:

1. **Splunk-side preparation**: create the default SC4S indexes, verify or
   create a HEC token, and validate the Cloud vs Enterprise HEC target.
2. **Runtime deployment**: render deployment assets for customer-managed SC4S
   infrastructure:
   - host-based Linux with Docker/Podman plus compose or systemd
   - Kubernetes with Helm

## Agent Behavior — Credentials

**The agent must NEVER ask for HEC tokens or other secrets in chat.**

- Splunk credentials come from the project-root `credentials` file or
  `~/.splunk/credentials`.
- Use `skills/splunk-connect-for-syslog-setup/template.example` as the
  non-secret intake worksheet.
- Keep HEC tokens in temporary or local-only files, for example:

```bash
bash skills/shared/scripts/write_secret_file.sh /tmp/sc4s_hec_token
```

- For Kubernetes, prefer a token-free `values.yaml` plus a local-only
  `values.secret.yaml` or an operator-managed install command. Do not commit HEC
  tokens to git.
- The default render path is the gitignored repo-local directory
  `./sc4s-rendered/`. When a real HEC token is being rendered, the setup script
  blocks custom output directories inside the repo and asks you to use the
  default gitignored path or a directory outside the repo.

If credentials are not configured yet:

```bash
bash skills/shared/scripts/setup_credentials.sh
```

## Environment

| Item | Value |
|------|-------|
| Search-tier API | `SPLUNK_SEARCH_API_URI` env var (legacy alias: `SPLUNK_URI`) |
| Cloud stack | `SPLUNK_CLOUD_STACK` for Splunk Cloud |
| Runtime image | `ghcr.io/splunk/splunk-connect-for-syslog/container3:latest` |
| Credentials | Project-root `credentials` file (falls back to `~/.splunk/credentials`) |
| Skill scripts | `skills/splunk-connect-for-syslog-setup/scripts/` |
| Templates | `skills/splunk-connect-for-syslog-setup/templates/` |

## Setup Workflow

### Step 1: Collect Non-Secret Deployment Inputs

Copy the worksheet locally:

```bash
cp skills/splunk-connect-for-syslog-setup/template.example template.local
```

Capture items such as:

- Splunk platform: Cloud or Enterprise
- deployment model: host or Kubernetes
- SC4S runtime root path
- HEC URL and HEC token name
- archive/TLS choices
- vendor/product dedicated listener ports
- optional context/config override files

### Step 2: Prepare Splunk

Create the SC4S indexes and verify or create a HEC token:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh --splunk-prep
```

Useful partial runs:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh --splunk-prep --indexes-only
```

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh --splunk-prep --hec-only
```

If you want the script to write the created token value to a local-only file
when Splunk REST returns it:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh \
  --splunk-prep \
  --write-hec-token-file /tmp/sc4s_hec_token
```

If Splunk cannot return that requested token value, preparation exits nonzero
and emits a rotate/create handoff; it does not report a usable token file.
Visible Selected Indexes restrictions must include every required SC4S index,
and HEC ACK must be disabled; otherwise preparation/validation return nonzero.

### Step 3: Render Host Deployment Assets

Compose example:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh \
  --render-host \
  --host-mode compose \
  --output-dir ./sc4s-rendered \
  --hec-token-file /tmp/sc4s_hec_token
```

The rendered compose directory is self-contained, so `--apply-host` can install
or upgrade it directly from `./sc4s-rendered/host/` without copying files into
`/opt/sc4s` first.

Systemd example:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh \
  --render-host \
  --host-mode systemd \
  --runtime podman \
  --output-dir ./sc4s-rendered \
  --sc4s-root /opt/sc4s \
  --hec-token-file /tmp/sc4s_hec_token
```

Systemd apply example:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh \
  --render-host \
  --host-mode systemd \
  --runtime podman \
  --output-dir ./sc4s-rendered \
  --sc4s-root /opt/sc4s \
  --hec-token-file /tmp/sc4s_hec_token \
  --apply-host
```

Optional dedicated vendor port:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh \
  --render-host \
  --output-dir ./sc4s-rendered \
  --hec-token-file /tmp/sc4s_hec_token \
  --vendor-port checkpoint:tcp:9000
```

### Step 4: Render Kubernetes Assets

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh \
  --render-k8s \
  --output-dir ./sc4s-rendered \
  --namespace sc4s \
  --release-name sc4s \
  --replica-count 2 \
  --hec-token-file /tmp/sc4s_hec_token
```

Optional context/config overrides:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh \
  --render-k8s \
  --output-dir ./sc4s-rendered \
  --hec-token-file /tmp/sc4s_hec_token \
  --context-file splunk_metadata.csv=/path/to/splunk_metadata.csv \
  --config-file app-workaround-cisco_asa.conf=/path/to/app-workaround-cisco_asa.conf
```

### Step 5: Optionally Apply Rendered Assets

Live host or Kubernetes apply requires a nonempty, owner-only
`--hec-token-file` (no group/other permission bits). A single
combined run may instead use `--splunk-prep --write-hec-token-file PATH`; apply
is blocked if preparation does not produce a nonempty token file.

Compose:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh \
  --render-host \
  --host-mode compose \
  --output-dir ./sc4s-rendered \
  --hec-token-file /tmp/sc4s_hec_token \
  --apply-host
```

Systemd:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh \
  --render-host \
  --host-mode systemd \
  --runtime podman \
  --output-dir ./sc4s-rendered \
  --sc4s-root /opt/sc4s \
  --hec-token-file /tmp/sc4s_hec_token \
  --apply-host
```

Helm:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/setup.sh \
  --render-k8s \
  --output-dir ./sc4s-rendered \
  --hec-token-file /tmp/sc4s_hec_token \
  --apply-k8s
```

### Step 6: Validate

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/validate.sh
```

Runtime-specific checks:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/validate.sh --check-host
```

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/validate.sh --check-k8s
```

## Key Learnings / Known Issues

1. **HEC ACK must stay off**: SC4S does not support HEC acknowledgement.
2. **Avoid index restrictions on the HEC token**: a restricted token can block
   batches when SC4S sends to an index not on the allow-list.
3. **Disabled HEC tokens are not treated as success**: the setup flow now
   enables an existing disabled token or fails clearly if it cannot.
4. **Disk buffer sizing matters**: the persistent volume can grow significantly
   during Splunk outages.
5. **Host networking differs by runtime**: compose typically publishes ports,
   while systemd examples often use host networking.
6. **Non-root Podman cannot bind 514/601**: if you need non-root operation, use
   alternate listener ports and update the sending devices accordingly.
7. **Kubernetes values differ from env_file settings**: use `values.yaml`,
   `context_files`, and `config_files` instead of copying host env_file layouts.
8. **`_metrics` must be a metrics index**: the setup flow now creates it as a
   metrics index and validation reports when an existing `_metrics` index was
   created with the wrong type.
9. **Re-run apply workflows for upgrades**: compose apply now pulls images
   before `up -d`, systemd apply syncs the rendered working set into
   `SC4S_ROOT`, and Kubernetes apply continues to use `helm upgrade --install`.

## Additional Resources

- [reference.md](reference.md) — index defaults, HEC behavior, deployment notes,
  and troubleshooting guidance
- [template.example](template.example) — non-secret intake worksheet
- [templates/host/README.md](templates/host/README.md) — host template notes
- [templates/kubernetes/README.md](templates/kubernetes/README.md) — Helm
  template notes

## MCP Tools

This skill includes checked-in, read-only Splunk MCP custom tools generated
from `mcp_tools.source.yaml`.

Validate or regenerate the tool artifact:

```bash
python3 skills/shared/scripts/mcp_tools.py validate skills/splunk-connect-for-syslog-setup
python3 skills/shared/scripts/mcp_tools.py generate skills/splunk-connect-for-syslog-setup
```

Load the tools into Splunk MCP Server:

```bash
bash skills/splunk-connect-for-syslog-setup/scripts/load_mcp_tools.sh
```

The loader uses the supported `/mcp_tools` REST batch endpoint by default. Use
`--allow-legacy-kv` only for older MCP Server app versions that lack that
endpoint.