Castai Common Errors

---
name: castai-common-errors
description: 'Diagnose and fix CAST AI agent, API, and autoscaler errors.

  Use when the CAST AI agent is offline, nodes are not scaling,

  or API calls return errors.

  Trigger with phrases like "cast ai error", "cast ai not working",

  "cast ai agent offline", "cast ai debug", "fix cast ai".

  '
allowed-tools: Read, Bash(kubectl:*), Bash(curl:*), Grep
version: 1.4.0
license: MIT
author: Jeremy Longshore <jeremy@intentsolutions.io>
tags:
- saas
- kubernetes
- cost-optimization
- castai
compatibility: Designed for Claude Code
---
# CAST AI Common Errors

## Overview

Diagnostic guide for the 10 most common CAST AI issues, covering agent connectivity, API errors, autoscaler failures, and node provisioning problems.

## Prerequisites

- `kubectl` access to the cluster
- `CASTAI_API_KEY` configured
- Access to CAST AI console for log correlation

## Instructions

Identify the affected cluster and environment first, collect the minimum
redacted evidence needed to classify the symptom, then use the matching
runbook below. Change one control at a time, observe health and workload impact
over the defined window, and keep production changes behind the approved
change process. Escalate rather than guessing when the issue involves
credentials, unexpected node termination, data loss risk, or an unclear
provider-side failure.

## Error Handling

| Failure class | Safe first action |
|---|---|
| Authentication or authorization | Stop retries, verify the scoped secret reference, and rotate through the owner if compromise is possible. |
| Autoscaler or evictor disruption | Disable the affected automation path and stabilize the workload before tuning. |
| Terraform drift or unsafe plan | Do not apply; reconcile through the declared state and review process. |
| Provider outage or unknown behavior | Preserve redacted timestamps and correlation details, then use the escalation path. |

## Error Reference

### 1. Agent Pod CrashLoopBackOff

```bash
kubectl get pods -n castai-agent
kubectl logs -n castai-agent deployment/castai-agent --tail=50
```

**Causes and fixes:**

- **Invalid API key**: Regenerate at console.cast.ai > API
- **Wrong provider**: Set `--set provider=eks|gke|aks` correctly in Helm
- **RBAC missing**: Apply the required ClusterRole and ClusterRoleBinding
- **Network blocked**: Ensure outbound HTTPS to `api.cast.ai` is allowed

### 2. Agent Shows "Disconnected" in Console

```bash
# Check agent heartbeat
kubectl logs -n castai-agent deployment/castai-agent | grep -i "heartbeat\|connect\|error"

# Verify network connectivity from inside the cluster
kubectl run castai-debug --image=curlimages/curl --rm -it --restart=Never -- \
  curl -s -o /dev/null -w "%{http_code}" https://api.cast.ai/v1/kubernetes/external-clusters
```

**Fix**: Restart the agent pod: `kubectl rollout restart deployment/castai-agent -n castai-agent`

### 3. API Returns 401 Unauthorized

```bash
# Test API key
curl -s -o /dev/null -w "%{http_code}" \
  -H "X-API-Key: ${CASTAI_API_KEY}" \
  https://api.cast.ai/v1/kubernetes/external-clusters
# Should return 200, not 401
```

**Fix**: Generate a new API key at console.cast.ai > API > API Access Keys.

### 4. Nodes Not Scaling Up (Unschedulable Pods)

```bash
# Check for pending pods
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

# Verify unschedulable pods policy is enabled
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
  | jq '.unschedulablePods'
```

**Causes:**

- `unschedulablePods.enabled` is `false` -- enable it
- Cluster limits reached -- increase `clusterLimits.cpu.maxCores`
- No matching node template -- check constraints match pod requirements

### 5. Nodes Not Scaling Down (Empty Nodes)

```bash
# Check node downscaler configuration
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
  | jq '.nodeDownscaler'
```

**Causes:**

- `nodeDownscaler.enabled` is `false`
- Pods with `PodDisruptionBudget` blocking eviction
- DaemonSet-only nodes with system pods preventing drain
- Delay too high -- reduce `emptyNodes.delaySeconds`

### 6. Spot Instance Fallback Not Working

```bash
# Check spot configuration
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
  | jq '.spotInstances'
```

**Fix**: Enable `spotDiversityEnabled: true` and set `spotDiversityPriceIncreaseLimitPercent` to 20-30 for better availability.

### 7. Evictor Too Aggressive

Symptoms: Pods being evicted too frequently, service disruption.

```bash
kubectl get events --field-selector reason=Evicted -A --sort-by=.lastTimestamp | tail -20
```

**Fix**: Increase evictor cycle interval or switch to non-aggressive mode:

```bash
helm upgrade castai-evictor castai-helm/castai-evictor \
  -n castai-agent \
  --set castai.apiKey="${CASTAI_API_KEY}" \
  --set castai.clusterID="${CASTAI_CLUSTER_ID}" \
  --set evictor.aggressiveMode=false \
  --set evictor.cycleInterval=600
```

### 8. Terraform State Drift

```bash
terraform plan -var-file=environments/prod.tfvars
# If drift detected:
terraform refresh -var-file=environments/prod.tfvars
```

**Fix**: Avoid mixing Terraform and console-based policy changes. Pick one source of truth.

### 9. Helm Chart Version Mismatch

```bash
# Check installed versions
helm list -n castai-agent
helm search repo castai-helm --versions | head -10

# Update to latest
helm repo update
helm upgrade castai-agent castai-helm/castai-agent -n castai-agent \
  --reuse-values
```

### 10. Workload Autoscaler Not Recommending

```bash
kubectl logs -n castai-agent deployment/castai-workload-autoscaler --tail=50
```

**Causes:**

- Insufficient metrics data (wait 24h)
- Missing annotation `autoscaling.cast.ai/enabled: "true"`
- Workload autoscaler pod not running

## Escalation Path

1. Collect debug info: Helm releases, agent logs, cluster events
2. Check https://status.cast.ai for platform issues
3. Contact support with cluster ID and screenshots

## Output

Produce a redacted incident record containing the affected cluster, symptom,
time window, observed evidence, containment action, selected runbook branch,
and escalation owner. Do not include API keys, raw secret values, or complete
customer workload logs; retain enough correlation information for support to
reproduce the provider-side investigation.

## Examples

For repeated evictions, capture the relevant event count and PDB state, disable
the aggressive evictor setting through the approved change path, and confirm
workload recovery before changing another variable. If the behavior persists,
attach the sanitized debug bundle and UTC incident window to the support case
instead of repeatedly restarting production components.

## Resources

- [CAST AI Troubleshooting](https://docs.cast.ai/docs/casti-ai-components)
- [CAST AI Status](https://status.cast.ai)
- [Autoscaler Checklist](https://docs.cast.ai/docs/autoscaler-checklist)

## Next Steps

For comprehensive diagnostics, see `castai-debug-bundle`.