Cisco Thousandeyes Setup

---
name: cisco-thousandeyes-setup
description: Use when configuring ThousandEyes OAuth, HEC, streaming or polling inputs, dashboards, or ITSI in Splunk.
compatibility: >-
  Splunk Cloud Platform 10.5.2605: conditional. Follow documented package,
  entitlement, topology, and customer-managed runtime guardrails; self-managed
  paths remain on the public 10.4 baseline.
metadata:
  splunk_cloud_10_5: "conditional"
  compatibility_verified: "2026-08-20"
---

# Cisco ThousandEyes App Setup Automation

## Prerequisites

| Tool or access | Purpose | Verify |
|---|---|---|
| Bash, `curl`, and `jq` | Run setup and REST configuration helpers | `command -v bash curl jq` |
| Splunk administrative access | Create indexes, HEC, accounts, and inputs | Confirm search-tier REST access |
| ThousandEyes account-group access | Authorize OAuth and selected test data | Record the numeric account-group ID |

## Workflow Overview

```text
┌───────────┐   ┌────────────────┐   ┌──────────────────┐   ┌────────────────────┐
│ Preflight │ → │ Configure auth │ → │ Enable data path │ → │ Validate dashboards │
└───────────┘   └────────────────┘   └──────────────────┘   └────────────────────┘
```

## When to Activate

- Configure ThousandEyes OAuth, HEC streaming, or API polling in Splunk.
- Onboard cloud, enterprise-agent, or endpoint test results.
- Diagnose missing path data, empty dashboards, or optional ITSI readiness.

## Scope

This skill configures the Splunk application and its data paths. It does not
create ThousandEyes tests, expose OAuth or HEC tokens in chat, or enable both
streaming and polling for the same data without duplicate-ingest review.

## Examples

Run preflight diagnostics before authentication:

```bash
bash skills/cisco-thousandeyes-setup/scripts/validate.sh
```

Expected output: package, local tools, Splunk access, and account prerequisites
are reported without creating tokens or inputs.

Run strict completion checks after enabling the selected path:

```bash
bash skills/cisco-thousandeyes-setup/scripts/validate.sh --completion
```

Expected output: account, HEC or polling input, index, source type, recent data,
and dashboard evidence report `[PASS]` or exit nonzero.

## Troubleshooting

| Issue | Cause | Resolution |
|---|---|---|
| Device flow expires | Approval timed out | Restart and approve promptly |
| No HEC events | Token, endpoint, or index is wrong | Verify without printing token |
| Duplicate test events | Streaming and polling overlap | Keep one path per dataset |
| Empty dashboards/ITSI | Index, macro, or type is wrong | Validate events first |

## TA Completion Gate

For every TA/add-on or dashboard companion run, satisfy the shared
[TA completion gate](../shared/ta_completion_gate.md): configure and enable the
data ingest path owned by this skill or its required companion, validate events
or metrics in the target indexes/source types, and verify any
pre-built/package-shipped dashboards are visible, macro-aligned, and returning
data. If the package ships no dashboards, record that evidence explicitly and
hand off dashboard use to the consuming app, ES/ITSI/ARI content, or readiness
doctor.

Automates the **Cisco ThousandEyes App for Splunk** (`ta_cisco_thousandeyes`).

## How This App Differs From Other Cisco TAs

This app is architecturally different from the Catalyst, Meraki, Intersight, and
DC Networking TAs in several ways:

- **OAuth 2.0 Device Code Flow** — no API keys or passwords. The user must
  visit a URL in their browser to authorize.
- **HEC-based data delivery** — ThousandEyes pushes most data TO Splunk via HEC
  (HTTP Event Collector) streams, rather than Splunk polling an API.
- **Hybrid collection** — 3 delivery mechanisms: streaming (HEC push), API
  polling, and webhooks.
- **Full app with dashboards** — not just a TA. Includes network, application,
  voice, alerts, traces, and configuration status dashboards.
- **ITSI integration** — optional bidirectional integration with Splunk IT
  Service Intelligence.

## Package Model

**Pull from Splunkbase first (latest version), fall back to `splunk-ta/`.**
Use `splunk-app-install` with `--source splunkbase --app-id 7719` to get the
latest release. If Splunkbase is unavailable, fall back to the local package
in `splunk-ta/`. This applies to both Splunk Cloud (ACS) and Splunk Enterprise.

After installation, use this skill to authenticate via OAuth, configure HEC,
create indexes, and enable inputs. Any `splunk-ta/_unpacked/` tree is
review-only.

## Agent Behavior — Credentials

**The agent must NEVER ask for passwords, API keys, or secrets in chat.**

Splunk credentials are read automatically from the project-root `credentials`
file (falls back to `~/.splunk/credentials`). If neither exists, guide the user
to create it:

```bash
bash skills/shared/scripts/setup_credentials.sh
```

### ThousandEyes OAuth Authentication

Unlike other TAs, this app uses an **interactive OAuth 2.0 device code flow**.
No password file or API key file is needed. Instead:

1. The `configure_account.sh` script initiates the OAuth flow and displays a
   **verification URL** and **user code**.
2. The agent instructs the user: "Visit the URL shown in your terminal and
   enter the code to authorize."
3. The script polls until the user completes authorization in their browser.
4. Tokens are stored and encrypted automatically by Splunk.

The agent never handles or sees the OAuth tokens.

For prerequisite collection, use `skills/cisco-thousandeyes-setup/template.example`
as the intake worksheet. Copy it to `template.local`, capture the non-secret
account details there, and keep the completed file local only.

## Environment

| Item | Value |
|------|-------|
| Search-tier API | `SPLUNK_SEARCH_API_URI` env var (legacy alias: `SPLUNK_URI`) |
| Cloud stack | `SPLUNK_CLOUD_STACK` for Cloud installs |
| App name | `ta_cisco_thousandeyes` |
| Credentials | Project-root `credentials` file (falls back to `~/.splunk/credentials`) |
| HEC requirement | Required for metrics, traces, activity logs, and alerts inputs |
| Skill scripts | `skills/cisco-thousandeyes-setup/scripts/` (relative to repo root) |

### Remote Splunk Connection

```bash
export SPLUNK_SEARCH_API_URI="https://splunk-host:8089"
```

## Splunk Authentication

Scripts read Splunk credentials from the project-root `credentials` file (falls
back to `~/.splunk/credentials`) automatically:

```bash
bash skills/cisco-thousandeyes-setup/scripts/validate.sh
```

If credentials are not yet configured:

```bash
bash skills/shared/scripts/setup_credentials.sh
```

## Setup Workflow

### Step 1: Install the App

```bash
bash skills/splunk-app-install/scripts/install_app.sh \
  --source splunkbase --app-id 7719
```

### Step 2: Create HEC Token and Indexes

```bash
bash skills/cisco-thousandeyes-setup/scripts/setup.sh
```

The default flow verifies or creates a HEC token named `thousandeyes`, then
creates six indexes. When run interactively (TTY), the script prompts to
continue with OAuth authentication and input enablement after the initial
setup completes.

For Splunk Cloud, HEC tokens are managed through ACS and indexes are created
through ACS. For Enterprise, both use the REST API.

Partial runs: `--hec-only`, `--indexes-only`.

Creates six indexes:

| Index | Purpose | Sourcetype |
|-------|---------|------------|
| `thousandeyes_metrics` | Test metrics data | `cisco:thousandeyes:metric` |
| `thousandeyes_traces` | Test traces data | `cisco:thousandeyes:trace` |
| `thousandeyes_events` | ThousandEyes events | `cisco:thousandeyes:event` |
| `thousandeyes_activity` | Activity/audit logs | `cisco:thousandeyes:activity` |
| `thousandeyes_alerts` | Alert notifications | `cisco:thousandeyes:alerts` |
| `thousandeyes_pathvis` | Path visualization | `cisco:thousandeyes:path-vis` |

The metrics and event collectors also write account-group metadata as
`cisco:thousandeyes:account-group` into their configured metrics or events
index. The token-refresh input writes operational logs to `_internal` as
`cisco:thousandeyes:refresh-tokens`; neither sourcetype needs a separate
customer index.

When you enable metrics inputs, the setup script now enables related path
collection into `thousandeyes_pathvis` by default. Although it is configured on
the metrics stream input, path visualization data is collected via API polling
on its own `--pathvis-interval` (default 3600s) — hence the "API polling"
delivery shown for `cisco:thousandeyes:path-vis` in the sourcetype table below.
Use `--no-pathvis` if you need a metrics-only stream, or override the defaults
with `--pathvis-index` / `--pathvis-interval`.

### Step 3: Authenticate via OAuth

```bash
bash skills/cisco-thousandeyes-setup/scripts/configure_account.sh
```

The script will:
1. Display a verification URL and user code
2. Wait for the user to authorize in their browser
3. Store the OAuth tokens in Splunk's encrypted credential store

The agent should instruct the user to visit the URL shown in the terminal
output and enter the displayed code.

### Step 4: Enable Inputs

Before running, the agent must ask the user for:
- ThousandEyes account name (the email shown after OAuth)
- ThousandEyes account group (ask user or use script to list available groups)
- Which input types to enable
- HEC token name (default: `thousandeyes`)
- Alert rule IDs when enabling `alerts` or `all`

```bash
bash skills/cisco-thousandeyes-setup/scripts/setup.sh --enable-inputs \
  --account "user@example.com" \
  --account-group "My Account Group" \
  --hec-token "thousandeyes" \
  --alert-rules "RULE_ID_1~RULE_ID_2" \
  --input-type all
```

| Input Type | Delivery | Description |
|------------|----------|-------------|
| `metrics` | HEC stream | Test metrics via ThousandEyes Streaming API |
| `traces` | HEC stream | Test traces via ThousandEyes Streaming API |
| `events` | API polling | ThousandEyes events (default interval: 3600s) |
| `activity` | HEC stream | Activity/audit logs via Streaming API |
| `alerts` | HEC webhook | Alert notifications via webhook |
| `all` | Mixed | All of the above |

The `alerts` and `all` groups require `--alert-rules` with ThousandEyes alert
rule IDs separated by `~`, matching the add-on's REST handler format.

### Step 5: ITSI Integration (Optional)

If Splunk ITSI (`SA-ITOA`) is installed, the app can forward Splunk notable
events to ThousandEyes and receive alert data. The validate script checks for
ITSI presence automatically.

### Step 6: Restart If Required

On Splunk Enterprise, restart Splunk after new index creation.
On Splunk Cloud, check `acs status current-stack` and only run
`acs restart current-stack` when ACS reports `restartRequired=true`.

### Step 7: Validate

```bash
bash skills/cisco-thousandeyes-setup/scripts/validate.sh --completion
```

Checks: app installation, HEC token, indexes, OAuth account, token refresh
input, data inputs, data flow, settings, and optional ITSI status.

## Sourcetypes

| Sourcetype | Delivery | Content |
|---|---|---|
| `cisco:thousandeyes:metric` | HEC stream | Test metrics (OpenTelemetry v2) |
| `cisco:thousandeyes:trace` | HEC stream | Test traces (OpenTelemetry v2) |
| `cisco:thousandeyes:path-vis` | API polling | Path visualization data |
| `cisco:thousandeyes:event` | API polling | ThousandEyes events |
| `cisco:thousandeyes:activity` | HEC stream | Activity/audit logs |
| `cisco:thousandeyes:alerts` | HEC webhook | Alert notifications |
| `cisco:thousandeyes:account-group` | API metadata | Account-group metadata written by metrics and event collectors |
| `cisco:thousandeyes:refresh-tokens` | Internal monitor | Token-refresh operational logs in `_internal` |

## Dashboards

The app ships dashboards in the package. They appear in Splunk Web
automatically after installation.

To access them: **Apps → Cisco ThousandEyes Add-On for Splunk**

Built-in dashboards include: Network, Application, Voice, Alerts, Traces, and
Configuration Status views.

**Prerequisites for dashboards to show data:**

1. The applicable `thousandeyes_*` indexes must exist and inputs must be
   enabled (Steps 2–4).
2. HEC must be reachable from ThousandEyes for streaming inputs (metrics,
   traces, activity, alerts). Validate the HEC target URL format — see
   Known Issue #6 below.
3. At least one polling input (`events`) or streaming input must have data
   flowing before dashboard panels populate.

On **Splunk Cloud**, the HEC URL must use the cloud format
(`https://http-inputs-{stack}.splunkcloud.com:443`). The setup script
auto-detects this when `SPLUNK_CLOUD_STACK` is set. If streaming inputs were
created with the wrong HEC URL, re-run `setup.sh --enable-inputs` to
correct them.

## MCP Server Integration

```bash
bash skills/cisco-thousandeyes-setup/scripts/load_mcp_tools.sh
```

## Key Learnings / Known Issues

1. **OAuth device code flow**: This is the only authentication method. There is
   no API key or username/password option. The user must complete authorization
   in a browser.
2. **HEC is required**: Metrics, traces, activity logs, and alerts all require a
   working HEC endpoint. Verify HEC is enabled and the token is valid before
   creating streaming inputs.
3. **Token refresh**: The app includes a `thousandeyes_refresh_tokens` modular
   input that runs weekly to regenerate OAuth tokens. Ensure this input is
   enabled.
4. **Streaming API creates ThousandEyes-side resources**: When you create a
   metrics or traces input, the app creates a "stream" object in ThousandEyes
   that pushes data to Splunk. Deleting the input should clean up the stream.
5. **Alerts use webhooks**: The alerts input creates webhook operations and
   connectors in ThousandEyes that push data to Splunk's HEC endpoint.
6. **Cloud vs Enterprise HEC URL format**: The HEC target URL differs by
   platform. Splunk Cloud uses `https://http-inputs-{stack}.splunkcloud.com:443`
   (or `.stg.splunkcloud.com` for staging). Enterprise uses
   `https://{host}:8088`. The `detect_hec_target()` function in `setup.sh`
   auto-detects the correct format, but falls back to the Enterprise pattern if
   `SPLUNK_CLOUD_STACK` is empty. If streaming inputs have a search-head
   hostname or port 8088 on a Cloud stack, ThousandEyes cannot reach the HEC
   endpoint and no data flows. The validate script checks for this mismatch.
   Fix by re-running `setup.sh --enable-inputs`.
7. **Client ID is hardcoded**: The OAuth client ID (`0oalgciz1dyS1Uonr697`) is
   built into the app; you do not need to provide it.
8. **ITSI is optional**: The ITSI integration only activates if `SA-ITOA` is
   installed. The app works fully without it.
9. **Existing account conflict**: If `configure_account.sh` fails with a
   409/500 ("Name already in use"), the account already exists from a prior
   session. The existing OAuth tokens are retained and typically still valid.
   Run `validate.sh` to check the account status before re-authenticating.
10. **Interactive continuation**: When run from a TTY, `setup.sh` prompts to
    continue with OAuth authentication and input enablement after HEC/index
    setup completes. This is skipped in non-interactive (piped) contexts.

## Additional Resources

- [reference.md](reference.md) — Complete input catalog, account fields, API
  endpoints, HEC configuration
- [mcp_tools.json](mcp_tools.json) — MCP tool definitions

## Validation Modes

Run `scripts/validate.sh` for diagnostics. Use `--completion` (alias `--strict`)
to require indexes, an OAuth account, token refresh, an enabled data input,
nonzero app data, expected account-group metadata when metrics or event data is
present, and visible shipped dashboards. A confirmed HEC token is required
when streaming inputs are present.