Splunk Observability Aws Integration

---
name: splunk-observability-aws-integration
description: "Use when the user asks to connect AWS to Splunk Observability Cloud, configure CloudWatch Metric
  Streams, render IAM policies, manage the AWSCloudWatch REST/Terraform object, monitor Bedrock metrics,
  set up multi-account AWS Organizations onboarding, audit drift, or migrate polling to Metric Streams.
  Hand off Lambda APM, AWS logs, dashboards, detectors, and EC2/EKS host telemetry to their owning skills.
  Render, preflight, apply, validate, discover, and diagnose the Splunk Observability Cloud AWSCloudWatch
  integration across polling, Splunk-managed Metric Streams, AWS-managed Metric Streams, and Terraform
  paths. Covers IAM trust and policy stubs, External ID and SecurityToken auth, CloudFormation and
  StackSets assets, Terraform payloads, field conflict checks, recommended stats, namespace sync rules,
  PrivateLink ingest stubs, drift adoption, and troubleshooting."
compatibility: "No direct Splunk Platform runtime dependency. This workflow can be used alongside Splunk Cloud Platform 10.5.2605 through its documented external APIs or handoffs."
metadata:
  splunk_cloud_10_5: "not-applicable"
  compatibility_verified: "2026-08-20"
---

# Splunk Observability Cloud <-> AWS Integration

## Prerequisites

| Tool or access | Purpose | Verify |
|---|---|---|
| Bash and Python 3 | Run bundled setup and validation helpers | `bash --version && python3 --version` |
| Required product/platform access | Inspect or configure the selected target | Complete the documented preflight |
| Credential files for live modes | Keep secrets out of chat | Verify paths only |

## Workflow Overview

```text
┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘
```

## When to Activate

- Connect AWS to Splunk Observability Cloud, configure CloudWatch Metric Streams, render IAM policies, manage the
  AWSCloudWatch REST/Terraform object, monitor Bedrock metrics, set up multi-account AWS Organizations onboarding,
  audit drift,.
- Preview and review the splunk observability aws integration workflow before any live apply phase.
- Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

## Scope

Follow the documented read-only or render-first path whenever it is available.
This skill does not imply permission to mutate live systems. Require explicit
apply flags, protected credentials, and operator review for state changes.

## Examples

Inspect the supported setup modes before selecting one:

```bash
bash skills/splunk-observability-aws-integration/scripts/setup.sh --help
```

Expected output: usage, supported modes, and required arguments are displayed
without changing the target environment.

Inspect validation modes before running completion checks:

```bash
bash skills/splunk-observability-aws-integration/scripts/validate.sh --help
```

Expected output: offline, live, and completion options are displayed when the
skill supports them; help exits without mutation.

## Troubleshooting

| Issue | Cause | Resolution |
|---|---|---|
| Preflight fails | A required tool or access path is missing | Resolve it before rendering or applying |
| Rendered assets are incomplete | Required non-secret inputs are absent | Complete intake and render again |
| Apply is blocked | Review, credentials, or explicit acceptance is missing | Use the documented handoff |
| Validation is incomplete | Live evidence is unavailable | Record the gap and keep completion open |

Render-first skill that owns the entire `AWSCloudWatch` integration object in
Splunk Observability Cloud. The skill is **standalone reusable** and does not
require any other Splunk Observability skill to run, but it hands off to other
skills for adjacent surfaces (logs, Lambda APM, dashboards, detectors, OTel
collectors).

The workflow is render-first by default. Live API changes only happen when the
user explicitly asks for `--apply`.

## Coverage Model

Every rendered section gets an explicit coverage status in
`coverage-report.json`:

- `api_apply` — a documented public REST API supports create / update / delete /
  validate (e.g. `POST /v2/integration`).
- `api_validate` — a documented public REST API supports read or validation only
  (e.g. preflighting `ec2:DescribeRegions` permission via the live integration's
  reported region list).
- `deeplink` — the skill renders a deterministic Splunk Observability Cloud UI
  link and validates referenced data where an API allows (e.g. the AWS
  guided-setup deeplink for AWS-managed Metric Streams).
- `handoff` — the skill renders deterministic operator steps for cross-skill
  workflows (e.g. logs path -> `splunk-app-install` for `Splunk_TA_AWS`).
- `not_applicable` — the section does not apply to the chosen target (e.g.
  Splunk-managed Metric Streams permissions when `connection.mode` is `polling`,
  or any AWS integration on `realm: us2-gcp` because that realm is GCP-hosted
  and has no AWS STS region mapping).

## Safety Rules

- Never ask for Splunk Observability tokens, AWS access keys, or AWS secret
  access keys in conversation.
- Never pass any secret on the command line or as an environment-variable
  prefix.
- Use `--token-file` for the Splunk Observability Cloud admin user API access
  token. Token files must be `chmod 600`. Override only with
  `--allow-loose-token-perms` (emits a WARN — use only for short-lived scratch
  tokens).
- Secret files must always be regular, non-symlink, single-hardlink files no
  larger than 64 KiB containing one non-empty printable-ASCII value with at
  most one trailing LF or CRLF. Readers use `O_NOFOLLOW`,
  bounded double reads, and stable metadata/content fingerprints;
  `--allow-loose-token-perms` relaxes only the mode check.
- Authenticated API calls are pinned to supported HTTPS realm hosts and refuse
  every redirect so `X-SF-Token` cannot be forwarded to another URL.
- For `authentication.mode: security_token` (GovCloud / China), use
  `--aws-access-key-id-file` and `--aws-secret-access-key-file`. Both files
  must be `chmod 600`.
- Reject direct secret flags: `--token`, `--access-token`, `--api-token`,
  `--o11y-token`, `--admin-token`, `--sf-token`, `--external-id`,
  `--aws-access-key-id`, `--aws-secret-access-key`, `--aws-secret-key`,
  `--password`.
- Prefer `SPLUNK_O11Y_REALM` and `SPLUNK_O11Y_TOKEN_FILE` from the repo
  `credentials` file when present; these store only realms and token-file
  paths, never token values.
- Strip every secret from `00-09-*.md`, `apply-plan.json`, `payloads/`,
  `current-state.json`, `state/apply-state.json`, and any other rendered
  artifact on disk. The Splunk Observability admin token is referenced as
  `${SPLUNK_O11Y_TOKEN_FILE}` everywhere, never inlined.
- The renderer FAILs render when the operator passes the deprecated
  `enableLogsSync` field with a clear pointer to the `Splunk_TA_AWS` handoff.
- The renderer FAILs render when `regions: []` (the canonical schema rejects an
  empty list and Splunk highly discourages it because new AWS regions
  auto-onboard and inflate cost).
- `bash skills/shared/scripts/write_secret_file.sh /tmp/splunk_o11y_token`
  helps the user create a token file without exposing the secret in shell
  history.

## Five-mode UX

| Mode | Flag | Purpose |
|------|------|---------|
| quickstart | `--quickstart` | Render the common scenario and print the supported SecurityToken apply plus ExternalId handoff. No live changes. |
| render | `--render` (default) | Produces the numbered plan tree under `--output-dir`. Never touches live state. |
| discover | `--discover` | Read-only sweep that polls `GET /v2/integration?type=AWSCloudWatch`, writes `current-state.json`, and emits a `drift-report.md` against the rendered plan. |
| doctor | `--doctor` | Runs the troubleshooting catalog and emits `doctor-report.md` with prioritized fixes and the exact `setup.sh --apply` command for each fix. |
| apply | `--apply [SECTIONS]` | Applies selected sections. SecurityToken integration and regional streams can mutate; IAM, ExternalId two-phase setup, and StackSets fail closed to explicit handoffs. |

Plus quality-of-life flags:

- `--quickstart-from-live` — turn the live integration into a `template.observed.yaml`
  the operator can edit; never overwrites their `template.example`.
- `--explain` — print the apply plan in plain English, no API calls; for
  change-management approvals.
- `--rollback <section>` — render (do not auto-run) the reverse-engineered
  commands for steps that have a public reversible API.
- `--list-namespaces` — print the supported AWS service / namespace catalog
  (mirrors `references/namespaces-catalog.md`).
- `--list-recommended-stats` — print the per-namespace per-metric stat catalog
  used by `collect_only_recommended_stats: true`.
- `--privatelink-domain {legacy,new}` — pick `signalfx.com` (default; matches
  current Splunk-published PrivateLink doc) vs `observability.splunkcloud.com`
  PrivateLink hostnames.
- `--cfn-template-url URL` — override the default
  `https://o11y-public.s3.amazonaws.com/aws-cloudformation-templates/release/template_metric_streams_regional.yaml`
  (or the StackSets equivalent when `metric_streams.use_stack_sets: true`).
  Overrides must be absolute HTTPS URLs on port 443, end in `.yaml` or `.yml`,
  and contain no credentials, query, fragment, whitespace, or control
  characters. The rendered AWS CLI command shell-quotes the validated URL.
- `--accept-drift FIELD[,FIELD...]` — needed for `--apply` when discover shows
  the live integration differs from the rendered spec on a field with side
  effects (e.g. flipping `useMetricStreamsSync`).

## Primary Workflow

1. Collect non-secret values: realm, integration name, AWS account ID(s),
   IAM role name, regions, services list (or `all_built_in`), connection
   mode (polling vs Splunk-managed Metric Streams vs AWS-managed Metric
   Streams), custom namespaces, multi-account toggle.

2. Create or update a JSON / YAML spec from `template.example`.

3. Render and validate:

   ```bash
   bash skills/splunk-observability-aws-integration/scripts/setup.sh \
     --render \
     --spec skills/splunk-observability-aws-integration/template.example \
     --output-dir splunk-observability-aws-integration-rendered
   ```

4. Review `splunk-observability-aws-integration-rendered/`:
   - `README.md` — TL;DR and ordered next-step commands.
   - `architecture.mmd` — Mermaid topology of the rendered integration.
   - `00-prerequisites.md` through `09-handoff.md` — numbered per-section plans.
   - `coverage-report.json` — per-section coverage status.
   - `apply-plan.json` — apply ordering with idempotency keys (no secrets).
   - `payloads/` — per-step request bodies for REST calls and CFN parameters.
   - `aws/` — CloudFormation template stubs (regional or StackSets) and
     Terraform `.tf` files for the AWS-side resources and the Splunk-side
     integration object.
   - `iam/` — per-use-case IAM JSON (foundation, polling, streams, tag-sync,
     Cassandra-special-case, GovCloud security-token).
   - `scripts/` — per-step apply scripts and cross-skill handoff drivers.
   - `support-tickets/` — pre-filled tickets when Splunk Support is required.

5. Apply SecurityToken mode only when explicitly requested. ExternalId mode
   uses the rendered two-phase handoff and direct apply refuses a partial
   disabled integration:

   ```bash
   bash skills/splunk-observability-aws-integration/scripts/setup.sh \
     --apply integration,validation \
     --spec skills/splunk-observability-aws-integration/template.example \
     --realm us1 \
     --token-file /tmp/splunk_o11y_admin_token \
     --aws-access-key-id-file /tmp/aws_access_key_id \
     --aws-secret-access-key-file /tmp/aws_secret_access_key
   ```

   To run only a subset of sections:

   ```bash
   bash skills/splunk-observability-aws-integration/scripts/setup.sh \
     --apply integration \
     --spec my-security-token-aws-integration.yaml \
     --token-file /tmp/splunk_o11y_admin_token \
     --aws-access-key-id-file /tmp/aws_access_key_id \
     --aws-secret-access-key-file /tmp/aws_secret_access_key
   ```

## Supported Sections

Specs use `api_version: splunk-observability-aws-integration/v1` and can
include the following top-level blocks (full reference in
[reference.md](reference.md)):

- `prerequisites` — region/realm preflight, FedRAMP/GovCloud/GCP carve-out,
  IAM permission check stubs, CFN template URL HTTP HEAD probe.
- `authentication` — `external_id` or `security_token` mode, AWS account ID,
  IAM role name, returned external ID.
- `connection` — polling vs Splunk-managed vs AWS-managed vs Terraform-only
  mode; `pollRate`, `metadataPollRate`, `inactiveMetricsPollRate`.
- `regions` — explicit AWS region list (cannot be empty).
- `services` — `all_built_in` / `explicit` / `namespace_filtered` /
  `custom_only`; `collect_only_recommended_stats`; `metric_stats_to_syncs`;
  `namespace_sync_rules` (for built-in `AWS/*` namespaces).
- `custom_namespaces` — `simple_list` (serializes to `customCloudwatchNamespaces`)
  OR `sync_rules` (serializes to `customNamespaceSyncRules`); they conflict.
  `sync_custom_namespaces_only` toggle.
- `guards` — `enable_check_large_volume`, `ignore_all_status_metrics`,
  `sync_load_balancer_target_group_tags`, `enable_aws_usage`.
- `metric_streams` — `use_metric_streams_sync`, `managed_externally`,
  `named_token`, `cloudformation`, `cloudformation_template_url`,
  `use_stack_sets`, `terraform`.
- `private_link` — `enable`, `endpoint_types`, `service_name_overrides`.
- `terraform_provider` — `source` (`splunk-terraform/signalfx`), `version`
  (`~> 9.0` default).
- `multi_account` — `enabled`, `control_account_id`, `member_accounts`,
  `cfn_stacksets`.
- `handoffs` — `lambda_apm`, `logs_via_splunk_ta_aws`, `dashboards`,
  `detectors`, `otel_collector_for_ec2_eks`.

## Hand-offs to Other Skills

- AWS log ingestion -> `bash skills/splunk-app-install/scripts/install_app.sh
  --source splunkbase --app-id 1876` (Splunk Add-on for AWS, Splunkbase 1876,
  min v8.1.1). Renderer preflights for `Splunk_TA_amazon_security_lake` and
  emits an uninstall step before the v7+ upgrade.
- Log Observer Connect to surface those logs in O11y ->
  [`splunk-observability-cloud-integration-setup`](../splunk-observability-cloud-integration-setup/SKILL.md).
- Native O11y dashboards for AWS namespaces ->
  [`splunk-observability-dashboard-builder`](../splunk-observability-dashboard-builder/SKILL.md).
- Detectors / AutoDetect alerting ->
  [`splunk-observability-native-ops`](../splunk-observability-native-ops/SKILL.md).
- OTel collector on EC2 / EKS for richer host telemetry than `CWAgent` ->
  [`splunk-observability-otel-collector-setup`](../splunk-observability-otel-collector-setup/SKILL.md).
- Lambda APM via the Splunk OpenTelemetry Lambda layer (publisher
  `254067382080`) ->
  [`splunk-observability-aws-lambda-apm-setup`](../splunk-observability-aws-lambda-apm-setup/SKILL.md).
  Renderer emits a hand-off stub in `09-handoff.md`.

## Out of Scope

- AWS log collection via the Splunk AWS log collector Lambda (handed off to
  `Splunk_TA_AWS`; the `enableLogsSync` API field is deprecated and rejected
  by the renderer).
- Lambda APM instrumentation via the OpenTelemetry Lambda layer (handed off to
  [`splunk-observability-aws-lambda-apm-setup`](../splunk-observability-aws-lambda-apm-setup/SKILL.md)).
- Native O11y AWS dashboard / detector CRUD (handed off to
  `splunk-observability-dashboard-builder` and
  `splunk-observability-native-ops`).
- AppDynamics for AWS workloads (separate AppDynamics SaaS workflow).
- Splunk Cloud Platform Data Manager AWS log onboarding (different product;
  Splunk Cloud Platform side, not Splunk Observability Cloud).

## Validation

```bash
bash skills/splunk-observability-aws-integration/scripts/validate.sh \
  --output-dir splunk-observability-aws-integration-rendered
```

Static checks: required-files, IAM JSON shape, and a secrets-leak scan across
every rendered file.

With `--live`, the CFN HTTPS probe is mandatory only when the rendered plan
uses that template. The live API check requires exactly one enabled integration
matching the rendered name, AWS account, regions, polling/stream mode, services,
custom namespaces, rates, and guards, then calls
`GET /v2/integration/validate/{id}`. Missing credentials, tools, or evidence
fail validation. The validation endpoint requires a short-lived administrator
User API session token.

For the cross-domain AWS/EKS/O11y production gate, render clean packets with
`scripts/staging/render-aws-eks-o11y.py` and run
`scripts/staging/validate-aws-eks-o11y.sh`. See
[`../../scripts/staging/README.md`](../../scripts/staging/README.md). The gate
is read-only and records a private, sanitized JSON report.

See `reference.md` and the focused references under `references/` for full
detail. The plan and corrections that produced this skill are recorded in
`.cursor/plans/splunk_o11y_aws_integration_skill_*.plan.md`.