Splunk Mcp Server Setup

---
name: splunk-mcp-server-setup
description: "Use when the user asks about Splunk MCP server setup, Splunk MCP TA, Splunk_MCP_Server, /services/mcp,
  the hosted SCS MCP Gateway for Splunk Observability Cloud, Cursor MCP, Codex MCP, Claude Code MCP
  connectivity to Splunk, or Cisco Data Fabric agentic/tool access through Splunk MCP. Install, configure,
  validate, and uninstall the Splunk MCP Server app for Cisco Data Fabric agentic/tool access
  (Splunk_MCP_Server / \"Splunk MCP TA\"). Configures mcp.conf server settings, rate limits, encrypted
  token issuance, and renders a shared client bridge bundle that works with Cursor, Codex, and Claude
  Code."
compatibility: "Splunk Cloud Platform 10.5.2605: conditional. Follow documented package, entitlement, topology, and customer-managed runtime guardrails; self-managed paths remain on the public 10.4 baseline."
metadata:
  splunk_cloud_10_5: "conditional"
  compatibility_verified: "2026-08-20"
---

# Splunk MCP Server Setup

## Prerequisites

| Tool or access | Purpose | Verify |
|---|---|---|
| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |
| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |
| Credential files for live modes | Keep secrets out of chat | Verify paths only |

## Workflow Overview

```text
┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘
```

## When to Activate

- Splunk MCP server setup, Splunk MCP TA, Splunk_MCP_Server, /services/mcp, the hosted SCS MCP Gateway for Splunk
  Observability Cloud, Cursor MCP, Codex MCP, Claude Code MCP connectivity to Splunk, or Cisco Data Fabric
  agentic/tool access.
- Preview and review the splunk mcp server setup workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

## Scope

Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.

## Examples

Inspect the supported setup modes before selecting one:

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh --help
```

Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.

Inspect validation modes before running completion checks:

```bash
bash skills/splunk-mcp-server-setup/scripts/validate.sh --help
```

Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.

## Troubleshooting

| Issue | Cause | Resolution |
|---|---|---|
| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |
| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |
| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |
| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |

## Production release status

The current official package, 1.3.1, is **not production-approved** by this
repository's security and protocol review. Installation fails closed by
default while vendor fixes are pending. Do not deploy it to production or
expose it to untrusted clients. `--accept-nonproduction-package` exists only
for isolated evaluation and does not make the package production-safe.

Version 1.3.1 is the supported evaluation baseline. It includes the 1.3
SPL2 `@spl2` query path, custom-tool UI, `find_data_source` preview, `rest`
command support, role-to-tool mappings, allowed-command administration, and
workload-pool selection. Treat those as vendor features: this skill validates
their shipped administration surfaces but does not silently enable tools,
expand allowed SPL commands, or assign roles.

## Shared add-on completion gate

Whenever this workflow installs, configures, or hands off the packaged MCP app,
follow the [shared completion gate](../shared/ta_completion_gate.md). Package
delivery alone is not success; validate the configured endpoint and its data
access, plus any shipped views, or record explicit package evidence that no
dashboards ship.

Automates setup of the **Splunk MCP Server** app (`Splunk_MCP_Server`).

For newer Cisco Data Fabric wording, this is the MCP/tool-access route for
agentic workflows. Federated search, data pipelines, and AI Toolkit workflows
remain in their dedicated skills.

## What This Skill Covers

This skill handles five operator tasks:

1. Install or update the packaged app from the repo-local `splunk-ta/` cache
2. Configure supported runtime settings in `mcp.conf`
3. Mint encrypted bearer tokens into local-only files
4. Render a reusable local bridge bundle for Cursor, Codex, and Claude Code,
   targeting either local Splunk Platform `/services/mcp` or the hosted SCS MCP Gateway
5. Uninstall the app cleanly when lab teardown is needed

The bridge bundle uses the same `mcp-remote` wrapper pattern for all three tools, so
one rendered directory can be opened in Cursor, registered with Codex, and auto-wired
into Claude Code's `.mcp.json`. The wrapper passes header placeholders such as
`${SPLUNK_MCP_HEADER_X_SF_TOKEN}` to `mcp-remote`, keeping token values in the
local env file instead of command argv.

`mcp-remote` is an experimental compatibility proxy. Prefer a client's native
Streamable HTTP transport when it can supply the required headers; the pinned
bridge is not itself a production trust boundary.

## Package Model

**Use the repo-local package in `splunk-ta/` as the default install source.**

The packaged app currently lives in:

```bash
splunk-ta/splunk-mcp-server_131.tgz
```

The setup workflow is the required install path because it enforces package
provenance and the production-review gate. For isolated evaluation only:

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --install \
  --accept-nonproduction-package
```

To remove the app again:

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh --uninstall
```

## Agent Behavior — Credentials And Tokens

**The agent must NEVER ask for passwords, bearer tokens, or other secrets in chat.**

Splunk credentials come from the project-root `credentials` file (falls back to
`~/.splunk/credentials`):

```bash
bash skills/shared/scripts/setup_credentials.sh
```

MCP bearer tokens are secrets. Always write them to a local-only file:

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --accept-nonproduction-package \
  --token-user "${SPLUNK_USER}" \
  --write-token-file /tmp/splunk_mcp.token
```

The agent may freely ask for non-secret values such as:
- MCP token username
- desired token lifetime
- row limits
- rate-limit thresholds
- whether the rendered client bridge should assume insecure TLS for lab use
- hosted SCS region, Observability realm, and Splunk tenant name

Use an existing Splunk user that has the `mcp_tool_admin` capability. In most
lab setups that should be the same account already configured in
`SPLUNK_USER`.

For prerequisite collection, use
`skills/splunk-mcp-server-setup/template.example` as the intake worksheet and
keep any filled copy local as `template.local`.

## Environment

| Item | Value |
|------|-------|
| Search-tier API | `SPLUNK_SEARCH_API_URI` env var (legacy alias: `SPLUNK_URI`) |
| Cloud stack | `SPLUNK_CLOUD_STACK` for Splunk Cloud |
| App name | `Splunk_MCP_Server` |
| Local package | `splunk-ta/splunk-mcp-server_131.tgz` (version 1.3.1; SHA-256 verified before install) |
| Credentials | Project-root `credentials` file (falls back to `~/.splunk/credentials`) |
| Skill scripts | `skills/splunk-mcp-server-setup/scripts/` |

## Setup Workflow

### Step 1: Install Or Update The App

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh --install
```

For an isolated Enterprise lab where service interruption should be deferred:

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --install \
  --no-restart \
  --accept-nonproduction-package
```

`--no-restart` only defers the shared installer restart; it does not guarantee
that updated Python handlers are active. Run validation afterward. If the live
endpoint still exposes the prior app behavior, restart Splunk before relying
on the upgrade.

This command intentionally refuses the current review-blocked release. After a
fixed vendor release is reviewed and marked production-approved in
`package-manifest.json`, the setup script will detect install versus update
mode automatically. Use `--accept-nonproduction-package` only in an isolated
lab.

### Alternative: Uninstall The App

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh --uninstall
```

This delegates to the shared app uninstaller for `Splunk_MCP_Server` and
restarts Splunk automatically on Enterprise targets unless the shared workflow
is changed to skip restart. Run `--uninstall` by itself; do not combine it with
render, token, or configuration flags.

### Step 2: Configure Supported MCP Server Settings

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --accept-nonproduction-package \
  --timeout 90 \
  --max-row-limit 2000 \
  --default-row-limit 250 \
  --ssl-verify true \
  --require-encrypted-token true \
  --legacy-token-grace-days 0 \
  --token-default-lifetime-seconds 43200 \
  --token-max-lifetime-seconds 86400 \
  --token-key-reload-interval-seconds 300 \
  --global-rate-limit 600 \
  --admission-global 60 \
  --tenant-authenticated 240 \
  --tenant-unauthenticated 10 \
  --circuit-breaker-failure-threshold 5 \
  --circuit-breaker-cooldown-seconds 60
```

This updates supported fields in `mcp.conf`:
- `[server] timeout`
- `[server] max_row_limit`
- `[server] default_row_limit`
- `[server] ssl_verify` (configuration intent only; vendor 1.3.1 does not
  enforce it for internal HTTP calls)
- `[server] require_encrypted_token`
- `[server] legacy_token_grace_days`
- `[server] mcp_token_default_lifetime_seconds`
- `[server] mcp_token_max_lifetime_seconds`
- `[server] token_key_reload_interval_seconds`
- `[rate_limits]` admission and circuit-breaker values

The script also fixes `visible=true` on the app if ACS or local installs left it
hidden in Splunk Web.

### Step 3: Optionally Rotate The MCP RSA Keys

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --accept-nonproduction-package \
  --rotate-keys \
  --rotate-key-size 4096
```

### Step 4: Mint An Encrypted Bearer Token

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --accept-nonproduction-package \
  --token-user "${SPLUNK_USER}" \
  --token-expires-on +12h \
  --write-token-file /tmp/splunk_mcp.token
```

The script writes the encrypted token to the target file with `0600`
permissions. It does not print the token to stdout.

Release 1.3.1 mints tokens with an authenticated `GET /mcp_token` request;
`POST /mcp_token` is reserved for `action=rotate`. The helper URL-encodes the
username and lifetime fields required by that vendor contract. Those non-secret
fields can appear in Splunk or intermediary access logs, but the returned token
is kept out of URLs and written only to the requested mode-`0600` file.

If you disable `require_encrypted_token`, the app intentionally fails closed on
`/mcp_token` minting and key rotation. Do not combine
`--require-encrypted-token false` with `--write-token-file` or `--rotate-keys`
in the same run.

### Step 5: Render And Apply The Shared Cursor/Codex Bridge Bundle

Choose one gateway mode:

| Mode | Endpoint | Required secret files |
|------|----------|-----------------------|
| `platform` | Splunk Platform app endpoint, usually `/services/mcp` on port `8089` | encrypted MCP bearer token file when writing a live `.env.splunk-mcp` |
| `o11y` | hosted SCS MCP Gateway | `--o11y-token-file` |
| `combined` | hosted SCS MCP Gateway with Splunk Platform + Observability headers | `--o11y-token-file` and `--splunk-jwt-file` |

Default platform mode preserves the existing local app workflow:

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --accept-nonproduction-package \
  --render-clients \
  --bearer-token-file /tmp/splunk_mcp.token \
  --cursor-workspace ~/Projects/my-cursor-workspace
```

O11y-only hosted gateway:

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --render-clients \
  --gateway-mode o11y \
  --scs-region pdx10 \
  --o11y-realm us1 \
  --o11y-token-file /tmp/splunk_o11y_api_token
```

Combined Splunk Platform + Observability gateway:

```bash
bash skills/splunk-mcp-server-setup/scripts/setup.sh \
  --render-clients \
  --gateway-mode combined \
  --scs-region pdx10 \
  --o11y-realm us1 \
  --o11y-token-file /tmp/splunk_o11y_api_token \
  --splunk-tenant mytenant \
  --splunk-jwt-file /tmp/splunk_mcp_jwt
```

The SCS gateway URL is derived as:

```text
https://region-<REGION>.api.scs.splunk.com/system/mcp-gateway/v1/
```

Current documented realm-to-SCS-region mappings:

| O11y realm | SCS region |
|------------|------------|
| `eu0` | `dub10` |
| `eu1` | `fra10` |
| `eu2` | `lon10` |
| `us0` | `iad10` |
| `us1` | `pdx10` |
| `us3` | `pdx10` |
| `jp0` | `tyo10` |
| `au0` | `syd10` |
| `sg0` | `sin10` |

Google Cloud Platform realms and GovCloud realms are not supported by the
hosted MCP Gateway; the renderer rejects known unsupported values such as
`us2` and `gov*`. Use `--gateway-url` only when Splunk provides an explicit
gateway endpoint.

Default render target:

```bash
./splunk-mcp-rendered
```

The rendered bundle contains:
- `.cursor/mcp.json` for Cursor
- `run-splunk-mcp.sh` as a shell stdio-to-HTTP bridge
- `run-splunk-mcp.js` as the Node bridge used by Cursor, Codex, and Claude Code registrations
- `.env.splunk-mcp.example`
- `.env.splunk-mcp` when a token file is supplied
- `register-codex-mcp.sh` to sync a portable Codex launcher bundle under `~/.codex/mcp-bridges/`

When `--render-clients` runs, the skill also applies client setup by default:
- registers `CLIENT_NAME` with Codex using a stable home-local launcher copy so repo moves do not break startup
- merges the Splunk MCP entry into `<cursor-workspace>/.cursor/mcp.json`
- writes the Splunk MCP entry into `<workspace>/.mcp.json` for Claude Code
- defaults the workspace target to the current working directory when
  `--cursor-workspace` is omitted

Use `--no-register-codex`, `--no-configure-cursor`, or `--no-configure-claude` to opt
out of any client update while still rendering the bundle.

Both wrappers require an operator-installed `mcp-remote@0.1.38` on `PATH`,
verify that exact package version, and fail closed rather than downloading code
at startup.
For `o11y` and `combined` gateway modes, the wrapper also passes
`--transport http-only --allow-http` to match Splunk's hosted gateway examples.

Do not add hosted Observability AI Assistant MCP tools to local
`Splunk_MCP_Server` custom tool manifests. Gateway mode only configures client
headers and endpoint selection.

### Step 6: Validate

```bash
bash skills/splunk-mcp-server-setup/scripts/validate.sh \
  --completion \
  --mcp-bearer-token-file /tmp/splunk_mcp.token
```

Checks:
- app installed and visible
- installed app version exactly matches the reviewed manifest (and is at least 1.3.1)
- `/services/mcp` completes authenticated `initialize`, `notifications/initialized`, `tools/list`, and a safe `splunk_get_info` tool call
- `/services/mcp` rejects an untrusted `Origin`
- enabled tools exactly match the reviewed allowlist (minimal default:
  `["splunk_get_info"]`; use `--allowed-tools-file` for a reviewed expansion)
- key MCP REST endpoints respond, including the 1.3 tool-role, guardrail, and
  allowed-SPL-command administration surfaces
- protected-resource metadata endpoint is reachable when configured
- current server settings and rate-limit values are readable
- encrypted-token, zero-grace, short-lifetime, and nonzero admission settings meet the production policy
- `ssl_verify` is reported as configuration-only and release 1.3.1 fails because the vendor does not enforce it
- derived `/services/mcp` URL is sane
- the shipped `dashboard`, `monitoring`, `tools`, and `tool_settings` views are visible

## Policy Surfaces In 1.3.1

The package still ships these baseline policy files with local-over-default
precedence:

- `local/safe_spl.json`
- `local/generating_commands.json`

Version 1.3.1 also ships authenticated administration endpoints for allowed SPL
commands, tool-role mappings, and guardrails. This skill validates that those
surfaces exist but does not mutate them. Review proposed commands and role
assignments separately, use the Splunk UI or documented app endpoint, and then
pass the exact enabled-tool policy to validation with `--allowed-tools-file`.
On self-managed targets, app-local overlays remain available when filesystem
ownership is explicit. On Splunk Cloud, treat direct file changes as package
content and use supported administration surfaces.

See [reference.md](reference.md) for the exact implications.

## Key Learnings / Known Issues

1. **`safe_spl.json` remains the local baseline**: the app loads it from the
   app directory; the 1.3 allowed-command endpoint manages a separate
   REST-backed command layer rather than rewriting that JSON file.
2. **Token output is secret material**: write encrypted bearer tokens to local
   files, never to chat or tracked repo files.
3. **The shared wrapper is the most portable client path**: Cursor, Codex, and
   Claude Code can all use the rendered `run-splunk-mcp.js` bridge via `mcp-remote`.
4. **`mcp.conf` is the supported remote configuration surface**: use it for
   row limits and token policy. Release 1.3.1 does not enforce its documented
   `ssl_verify` value, so that field cannot satisfy a TLS control.
5. **The app needs search-tier placement**: it exposes `/services/mcp` and
   depends on custom REST handlers plus KV Store-backed tool metadata.
6. **Hosted SCS MCP Gateway is client-side configuration**: it uses
   `--gateway-mode o11y` or `combined` and does not install hosted
   Observability tools into the local Splunk Platform app.
7. **Safe-SPL exclusion is defective in 1.3.1**: never use `exclude_tools` to
   disable a tool; use `mcp_tools_enabled`, and keep query tools away from
   untrusted callers pending a vendor fix.
8. **Evaluation data must be synthetic**: 1.3.1 logs tool arguments and SPL to
   `_internal`. Never embed literal credentials in custom tool headers or bodies.

## Client Activation

For Cursor and Claude Code activation details, including the local-only token
file and reload requirements, read the [client activation notes](reference.md#client-activation).

## Additional Resources

- [reference.md](reference.md) — endpoint map, config surface, and client notes
- [template.example](template.example) — non-secret intake worksheet