Api Latency Investigation

---
name: api-latency-investigation
enabled: true
description: |
  Use when performing api latency investigation — provides a systematic
  investigation framework for diagnosing and resolving API latency issues.
  Covers distributed tracing analysis, bottleneck identification across the
  request path, database query impact, external dependency latency, and
  optimization recommendations with expected impact.
required_connections:
  - prefix: apm
    label: "APM / Tracing Platform"
  - prefix: monitoring
    label: "Monitoring Platform"
config_fields:
  - key: api_endpoint
    label: "Affected API Endpoint(s)"
    required: true
    placeholder: "e.g., GET /api/v1/orders, POST /api/v1/checkout"
  - key: current_latency
    label: "Current P95 Latency"
    required: true
    placeholder: "e.g., 2500ms"
  - key: target_latency
    label: "Target P95 Latency"
    required: false
    placeholder: "e.g., 500ms"
features:
  - PERFORMANCE
  - API
  - INVESTIGATION
---

# API Latency Investigation

## Phase 1: Symptom Characterization
1. Define the latency problem precisely
   - [ ] Which endpoints are affected?
   - [ ] When did latency increase? (gradual or sudden)
   - [ ] Is it constant or intermittent?
   - [ ] Does it correlate with traffic volume?
   - [ ] Which percentiles are affected? (P50, P95, P99)
   - [ ] Are specific users, regions, or clients affected?
2. Collect current latency metrics
3. Compare against historical baseline
4. Check for recent deployments or infrastructure changes

### Latency Profile

| Endpoint | P50 (current) | P50 (baseline) | P95 (current) | P95 (baseline) | P99 (current) |
|----------|-------------|---------------|-------------|---------------|-------------|
|          | ms          | ms            | ms          | ms            | ms          |

## Phase 2: Request Path Analysis
1. Trace the full request path
   - [ ] Client / CDN / Load Balancer
   - [ ] API Gateway / Reverse Proxy
   - [ ] Application server processing
   - [ ] Database queries
   - [ ] Cache lookups (hit/miss)
   - [ ] External API calls
   - [ ] Message queue operations
   - [ ] Response serialization
2. Measure time spent at each component using distributed tracing
3. Identify the component consuming most time

### Latency Breakdown

| Component | Avg Time (ms) | % of Total | Variance | Bottleneck |
|-----------|-------------|-----------|----------|------------|
| Network/LB | | % | Low/High | [ ] |
| API Gateway | | % | | [ ] |
| App processing | | % | | [ ] |
| Database | | % | | [ ] |
| Cache | | % | | [ ] |
| External APIs | | % | | [ ] |
| Serialization | | % | | [ ] |
| **Total** | | **100%** | | |

## Phase 3: Bottleneck Deep Dive
1. For database bottlenecks:
   - [ ] Review slow queries and execution plans
   - [ ] Check for missing indexes
   - [ ] Analyze connection pool utilization
   - [ ] Check for lock contention
2. For application bottlenecks:
   - [ ] Profile CPU hotspots
   - [ ] Check for memory pressure / GC pauses
   - [ ] Review thread/goroutine blocking
   - [ ] Check for synchronous operations that should be async
3. For external dependency bottlenecks:
   - [ ] Measure dependency response times
   - [ ] Check for timeout configurations
   - [ ] Evaluate circuit breaker behavior
   - [ ] Consider caching dependency responses
4. For network bottlenecks:
   - [ ] Check DNS resolution times
   - [ ] Verify TLS handshake performance
   - [ ] Measure cross-AZ/region latency
   - [ ] Review connection reuse and keep-alive settings

## Phase 4: Root Cause Identification
1. Correlate latency with system metrics
2. Check for resource saturation (CPU, memory, connections, IOPS)
3. Review application logs for errors and warnings
4. Identify any cascade effects from downstream services
5. Document confirmed root cause(s)

### Root Cause Analysis

| Root Cause | Evidence | Impact on Latency | Fix Complexity | Priority |
|-----------|----------|-------------------|---------------|----------|
|           |          | +ms              | Low/Med/High  | 1-5      |

## Phase 5: Optimization Implementation
1. Implement fixes in priority order
   - [ ] Database query optimization and indexing
   - [ ] Caching layer (application cache, Redis, CDN)
   - [ ] Connection pooling tuning
   - [ ] Async processing for non-critical paths
   - [ ] Payload size reduction
   - [ ] Batch/bulk API operations
   - [ ] Circuit breakers for external dependencies
2. Test each fix and measure latency impact
3. Verify no regressions introduced

## Phase 6: Validation & Monitoring
1. Compare post-fix latency against baseline
2. Validate across all affected percentiles
3. Run load test at peak traffic levels
4. Set up latency alerting and SLO monitoring
5. Document findings for future reference

## Counter-Rationalizations

| Shortcut | Counter | Why |
|----------|---------|-----|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |

## Output Format
- **Latency Profile**: Current vs. baseline by percentile
- **Request Path Breakdown**: Time per component analysis
- **Root Cause Report**: Confirmed causes with evidence
- **Optimization Results**: Before/after latency measurements
- **Monitoring Setup**: Alerts and dashboards for ongoing tracking

## Action Items
- [ ] Characterize the latency problem with precise metrics
- [ ] Trace request path and identify bottleneck components
- [ ] Deep dive into top bottleneck(s)
- [ ] Implement optimizations in priority order
- [ ] Validate latency improvement against targets
- [ ] Set up ongoing latency monitoring and alerting