Durable Agent Execution Expert

---
name: durable-agent-execution-expert
description: "Durable agent execution, Temporal AI workflows, and state replay / Eksekusi agen AI tahan-gagal, workflow Temporal, dan replay state."
author: "Roedy Rustam"
version: "4.8.0"

---

# Durable Agent Execution & Crash-Resilient Workflow Expert (2026 Edition)

[English](#english) | [Bahasa Indonesia](#bahasa-indonesia)

---

<a name="english"></a>
## English

### Orchestration & Integration
- **Async Queues & Distributed Sagas**: `async-queue-temporal-expert`, `event-driven-architect`, `error-resilience-expert`
- **AI Agents & Orchestration**: `multi-agent-orchestration`, `ai-llm-integration-expert`, `a2a-agent-protocol-expert`, `agentic-memory-architect`
- **Edge & Serverless Runtimes**: `cloudflare-workers-expert`, `js-backend-expert`
- **Lifecycle & Governance**: `brainstorming`, `zero-to-prod-orchestrator`, `session-memory-manager`

### Description
Production engineering guide for building **crash-proof, long-running AI Agents** using **Durable Execution Engines** (**Temporal.io**, **Restate**, **Cloudflare Workflows**, and **LangGraph Durable Checkpointers**). Standard HTTP handlers fail when an LLM loop takes 5 minutes, hits a rate-limit, or waits 48 hours for human approval. Durable execution journals every LLM call and tool result so if a worker container crashes at step 9 of 10, it resumes immediately at step 9 without re-billing tokens or re-executing side effects.

### Trigger Conditions
- Building autonomous coding, research, or FinTech reconciliation agents that run for minutes, hours, or days.
- Implementing Human-in-the-Loop (HITL) approval gates where an agent pauses execution without holding open a server thread.
- Preventing duplicate LLM token charges or duplicate external API calls when a worker restarts mid-loop.
- Deploying stateful multi-step workflows on Cloudflare Workflows or Restate.

### Hard Guardrails (Declarative Invariants)
- **Wrap All Non-Deterministic Operations in Journaled Steps**: Never invoke LLM APIs, `fetch()`, `Date.now()`, `Math.random()`, or `crypto.randomUUID()` directly inside a workflow orchestrator body; wrap every side effect inside `step.do()` or a Temporal Activity.
- **Deterministic Step Idempotency Keys**: Pass `${workflowId}:${stepName}` as the `Idempotency-Key` header on every external mutation tool call so activity retries never double-charge or double-post.
- **Zero-Compute HITL Signals (`waitForEvent`)**: Pause for human approval using durable events (`step.waitForEvent` or Temporal Signals + `condition()`) with an explicit timeout—never `setInterval` or busy-wait loops.
- **Bound Journal Payload Sizes (< 512 KB)**: Store large PDFs, screenshots, or embeddings in S3/R2 and pass only content-addressed URIs + SHA-256 digests through the workflow journal.
- **Explicit Retry & Non-Retryable Error Classification**: Configure exponential backoff for transient `429`/`503` LLM errors while marking schema validation (`400`) and auth (`401`/`403`) errors as non-retryable (`ApplicationFailure.nonRetryable`).
- **Token Budget & Max-Iteration Circuit Breaker**: Enforce a hard `maxIterations` and cumulative token cost ceiling inside the durable workflow state to prevent runaway agent loops.

---

### Durable Engine Selection Matrix

| Engine | Runtime Model | State Storage | Best For |
| :--- | :--- | :--- | :--- |
| **Temporal.io** | Long-polling workers (Node/Go/Python/Rust) | Postgres / Cassandra | Complex enterprise sagas, multi-day agent loops |
| **Restate** | Push/RPC Durable Invocations (TS/Rust/Go) | Built-in RocksDB / Raft log | Low-latency microservices, durable idempotency & virtual objects |
| **Cloudflare Workflows** | Serverless Edge (`WorkflowEntrypoint`) | Cloudflare Durable Storage | Zero-infra edge agents, `step.do()` & `step.sleep()` |
| **LangGraph Checkpointer** | Graph State Snapshots | Postgres (`PostgresSaver`) | Conversational time-travel, state branching, thread resume |

---

### Core Implementation Patterns

#### 1. Cloudflare Workflows Durable Agent Loop (`step.do` & `step.waitForEvent`)
Every non-deterministic operation (LLM generation, external tool fetch, `Date.now()`) MUST be wrapped inside a durable `step.do()` block so its return value is persisted in the execution journal:

```typescript
// src/workflows/durable-research-agent.ts
export interface ResearchJobParams {
  jobId: string;
  tenantId: string;
  topic: string;
  requireHumanSignoff: boolean;
}

export interface WorkflowStep {
  do<T>(name: string, config: { retries?: { limit: number; delay: string; backoff: 'exponential' } }, fn: () => Promise<T>): Promise<T>;
  sleep(name: string, duration: string): Promise<void>;
  waitForEvent<T>(name: string, options: { type: string; timeout: string }): Promise<{ payload: T }>;
}

export async function runDurableAgentWorkflow(
  params: ResearchJobParams,
  step: WorkflowStep
) {
  // Step 1: Journaled LLM planning call (never re-runs on downstream retry)
  const plan = await step.do(
    'generate-research-plan',
    { retries: { limit: 3, delay: '2 seconds', backoff: 'exponential' } },
    async () => {
      return {
        queries: [`${params.topic} architecture 2026`, `${params.topic} security invariants`],
        estimatedTokens: 4096,
      };
    }
  );

  // Step 2: Journaled parallel evidence gathering
  const synthesis = await step.do(
    'synthesize-findings',
    { retries: { limit: 3, delay: '5 seconds', backoff: 'exponential' } },
    async () => {
      return {
        summary: `Verified findings for ${params.topic}`,
        sourcesCount: plan.queries.length,
      };
    }
  );

  // Step 3: Zero-compute durable pause for Human-in-the-Loop approval (up to 72 hours)
  if (params.requireHumanSignoff) {
    const approval = await step.waitForEvent<{ approved: boolean; reviewerId: string }>(
      'await-human-approval',
      { type: 'human.approval.submitted', timeout: '72 hours' }
    );

    if (!approval.payload.approved) {
      return { status: 'REJECTED_BY_REVIEWER' as const, reviewerId: approval.payload.reviewerId };
    }
  }

  // Step 4: Commit final output
  return await step.do('publish-artifact', {}, async () => {
    return {
      status: 'COMPLETED' as const,
      jobId: params.jobId,
      synthesis,
    };
  });
}
```

---

### Verification Checklist
- [ ] All non-deterministic calls (`fetch`, LLM inference, `crypto.randomUUID()`, `Date.now()`) are executed inside journaled `step.do()` / Activity boundaries.
- [ ] External side-effect tools pass deterministic idempotency keys (`${workflowId}:${stepName}`) to downstream APIs.
- [ ] Human-in-the-Loop pauses use durable signals/events (`waitForEvent` / Temporal `defineSignal`) with explicit timeout fallbacks rather than `setInterval` polling.
- [ ] Workflow state payloads remain compact (< 512 KB); large binary artifacts are stored in S3/R2 and referenced by URI.

---

#### Idempotent Step Checkpoint Executor (TypeScript)
```typescript
export interface StepJournalStore {
  getStepOutput<T>(workflowId: string, stepKey: string): Promise<T | undefined>;
  saveStepOutput<T>(workflowId: string, stepKey: string, output: T): Promise<void>;
}

export async function runDurableStep<T>(
  store: StepJournalStore,
  workflowId: string,
  stepKey: string,
  operation: () => Promise<T>
): Promise<T> {
  const cached = await store.getStepOutput<T>(workflowId, stepKey);
  if (cached !== undefined) {
    return cached;
  }
  const freshOutput = await operation();
  await store.saveStepOutput(workflowId, stepKey, freshOutput);
  return freshOutput;
}
```

---

#### 3. Temporal.io AI Agent Saga Workflow with Signal Handlers (TypeScript)
Enterprise sagas with multi-day durability and real-time operator overrides:

```typescript
// src/workflows/temporal-agent-saga.ts
export interface AgentSagaInput {
  workflowId: string;
  tenantId: string;
  maxIterations: number;
  initialPrompt: string;
}

export interface AgentSignalPayload {
  action: 'PAUSE' | 'RESUME' | 'INJECT_CORRECTION' | 'ABORT';
  message?: string;
}

export class TemporalAgentWorkflowSaga {
  private isPaused = false;
  private isAborted = false;
  private pendingCorrection: string | null = null;
  private currentIteration = 0;

  public handleSignal(payload: AgentSignalPayload): void {
    if (payload.action === 'PAUSE') {
      this.isPaused = true;
    } else if (payload.action === 'RESUME') {
      this.isPaused = false;
    } else if (payload.action === 'ABORT') {
      this.isAborted = true;
    } else if (payload.action === 'INJECT_CORRECTION' && payload.message) {
      this.pendingCorrection = payload.message;
    }
  }

  public async execute(
    input: AgentSagaInput,
    activities: {
      invokeLlmStep: (prompt: string, iteration: number) => Promise<{ nextAction: string; isComplete: boolean }>;
      executeToolCall: (action: string) => Promise<{ result: string }>;
      publishResult: (workflowId: string, output: string) => Promise<void>;
    }
  ): Promise<{ status: 'COMPLETED' | 'ABORTED'; iterations: number }> {
    let currentContext = input.initialPrompt;

    while (this.currentIteration < input.maxIterations) {
      if (this.isAborted) {
        return { status: 'ABORTED', iterations: this.currentIteration };
      }

      if (this.isPaused) {
        // Yield execution without burning CPU or memory
        await new Promise((resolve) => setTimeout(resolve, 1000));
        continue;
      }

      if (this.pendingCorrection) {
        currentContext += `\n[Operator Correction]: ${this.pendingCorrection}`;
        this.pendingCorrection = null;
      }

      this.currentIteration += 1;
      const stepResult = await activities.invokeLlmStep(currentContext, this.currentIteration);

      if (stepResult.isComplete) {
        await activities.publishResult(input.workflowId, stepResult.nextAction);
        return { status: 'COMPLETED', iterations: this.currentIteration };
      }

      const toolResult = await activities.executeToolCall(stepResult.nextAction);
      currentContext += `\n[Tool Output]: ${toolResult.result}`;
    }

    await activities.publishResult(input.workflowId, currentContext);
    return { status: 'COMPLETED', iterations: this.currentIteration };
  }
}
```

#### 4. Restate.dev Durable Virtual Object Agent with Awakeable HITL Gate (TypeScript)
Persist per-session agent state and pause on cryptographic human sign-off using Restate `ctx.run` and `ctx.awakeable`:

```typescript
// src/workflows/restate-agent-object.ts
export interface RestateObjectContext {
  key: string;
  get<T>(stateKey: string): Promise<T | null>;
  set<T>(stateKey: string, value: T): void;
  run<T>(name: string, fn: () => Promise<T>): Promise<T>;
  awakeable<T>(): { id: string; promise: Promise<T> };
  resolveAwakeable<T>(id: string, payload: T): void;
}

export interface PayoutProposal {
  proposalId: string;
  amountMinor: number;
  currency: 'IDR' | 'USD';
  recipientAccount: string;
}

export async function executeDurablePayoutAgent(
  ctx: RestateObjectContext,
  proposal: PayoutProposal
): Promise<{ status: 'EXECUTED' | 'DENIED'; awakeableId: string; txRef?: string }> {
  const riskScore = await ctx.run('score-fraud-risk', async () => {
    return proposal.amountMinor > 10_000_000 ? 85 : 10;
  });

  const gate = ctx.awakeable<{ approved: boolean; approverId: string }>();
  ctx.set('pending_awakeable_id', gate.id);

  if (riskScore >= 50) {
    const decision = await gate.promise;
    if (!decision.approved) {
      return { status: 'DENIED', awakeableId: gate.id };
    }
  }

  const txRef = await ctx.run('execute-ledger-transfer', async () => {
    return `tx_${ctx.key}_${proposal.proposalId}`;
  });

  return { status: 'EXECUTED', awakeableId: gate.id, txRef };
}
```

#### 5. Durable Token Budget & Cost Circuit Breaker (TypeScript)
Halt multi-step durable agent workflows deterministically before exceeding tenant FinOps budgets:

```typescript
// src/workflows/durable-budget-breaker.ts
export interface WorkflowCostLedger {
  readonly workflowId: string;
  readonly maxBudgetMicrosUsd: number;
  readonly spentMicrosUsd: number;
  readonly stepCount: number;
  readonly maxSteps: number;
}

export function recordDurableStepCost(
  ledger: WorkflowCostLedger,
  stepCostMicrosUsd: number
): WorkflowCostLedger {
  const nextSpent = ledger.spentMicrosUsd + stepCostMicrosUsd;
  const nextStep = ledger.stepCount + 1;

  if (nextStep > ledger.maxSteps) {
    throw new Error(`[BudgetBreaker] Workflow ${ledger.workflowId} exceeded maxSteps (${ledger.maxSteps})`);
  }
  if (nextSpent > ledger.maxBudgetMicrosUsd) {
    throw new Error(
      `[BudgetBreaker] Workflow ${ledger.workflowId} exceeded budget: ${nextSpent} > ${ledger.maxBudgetMicrosUsd} micros USD`
    );
  }

  return {
    ...ledger,
    spentMicrosUsd: nextSpent,
    stepCount: nextStep,
  };
}
```

---

### Sovereign Execution, Token Efficiency & Invariants
| Dimension | Sovereign Mandate & Deterministic Contract |
| :--- | :--- |
| **Frontier Model Cascade** | Medium / High (`GPT-5.6 Sol Medium`, `Gemini 4 Argon`, `Claude Sonnet 5.5 High`, `Grok 4.7 High Fast`) |
| **Availability Fallback** | `Gemini 4 Argon/Pro` -> `Gemini 3.8 Flash`; `Claude Opus 5.5` -> `Claude Opus 4.6` (Zero unhandled missing-model errors) |
| **KV-Cache & Token Protocol** | Pin static instructions & schemas at prompt prefix (`kv-cache-prefix-optimizer`); pass surgical `CONTEXT_MAP.md` slices (<250 tokens/call) during Swarm Fan-Out |
| **SaaT CLI & MCP Dispatch** | CLI: `vibes skill exec durable-agent-execution-expert <audit\|eval\|verify\|generate\|blueprint>` / MCP: `vibes_skill_dispatch`, `vibes_ast_query`, `vibes_compile_cache`, `vibes_optimize_budget`, `vibes_verify_gate` |
| **NFR & Anti-Slop Invariants** | P99 API Gateway Latency < 85ms, OpenAPI 3.1 / TypeSpec strict schema parity, RFC 9457 Problem Details error contracts, Idempotency-Key enforcement; 100% complete zero-placeholder output verified via `node scripts/check-anti-slop.js --strict` |

---


---

<a name="bahasa-indonesia"></a>
## Bahasa Indonesia

### Integrasi Orkestrasi
- **Antrean Asinkron & Saga Terdistribusi**: `async-queue-temporal-expert`, `event-driven-architect`, `error-resilience-expert`
- **Agen AI & Orkestrasi**: `multi-agent-orchestration`, `ai-llm-integration-expert`, `a2a-agent-protocol-expert`, `agentic-memory-architect`
- **Runtime Edge & Serverless**: `cloudflare-workers-expert`, `js-backend-expert`
- **Siklus Hidup & Tata Kelola**: `brainstorming`, `zero-to-prod-orchestrator`, `session-memory-manager`

### Deskripsi
Panduan rekayasa produksi untuk membangun **Agen AI berdurasi panjang yang tahan kegagalan (*crash-resilient*)** menggunakan **Temporal.io**, **Restate**, **Cloudflare Workflows**, dan **LangGraph Durable Checkpointers**. Setiap pemanggilan LLM dan tool dicatat dalam jurnal (*execution log*), sehingga jika container restart di tengah jalan, agen melanjutkan tepat dari langkah terakhir tanpa menagih ulang token LLM.

### Panduan Arsitektur Inti
1. **Determinisme Workflow**: Bungkus setiap pemanggilan LLM, I/O jaringan, dan waktu sistem di dalam blok `step.do()` atau *Activity* agar hasilnya disimpan di jurnal.
2. **Human-in-the-Loop Tanpa Biaya Compute**: Gunakan `step.waitForEvent()` atau Temporal *Signal* untuk menjeda agen selama berjam-jam menunggu persetujuan manusia tanpa menahan koneksi memori server.
3. **Kunci Idempotensi Langkah**: Sertakan `${workflowId}:${stepName}` sebagai kunci idempotensi saat memanggil tool eksternal.

### Daftar Periksa Verifikasi (Checklist)
- [ ] Seluruh pemanggilan LLM dan I/O dibungkus dalam *durable step/activity*.
- [ ] Tool dengan efek samping menggunakan kunci idempotensi deterministik.
- [ ] Jeda *human-in-the-loop* memiliki batas waktu (*timeout*) eksplisit.

### Matriks Eksekusi Berdaulat, Efisiensi Token & Invarian
| Dimensi | Mandat Berdaulat & Kontrak Deterministik |
| :--- | :--- |
| **Kaskade Model Frontier** | Medium / High (`GPT-5.6 Sol Medium`, `Gemini 4 Argon`, `Claude Sonnet 5.5 High`, `Grok 4.7 High Fast`) |
| **Invarian Fallback Ketersediaan** | `Gemini 4 Argon/Pro` -> `Gemini 3.8 Flash`; `Claude Opus 5.5` -> `Claude Opus 4.6` (Bebas gagal saat model belum tersedia/kuota penuh) |
| **Protokol Hemat Token & KV-Cache** | Kunci instruksi & skema statis di prefix awal (`kv-cache-prefix-optimizer`); kirim potongan `CONTEXT_MAP.md` (<250 token/panggilan) saat Fan-Out |
| **Eksekusi SaaT CLI & MCP** | CLI: `vibes skill exec durable-agent-execution-expert <audit\|eval\|verify\|generate\|blueprint>` / MCP: `vibes_skill_dispatch`, `vibes_ast_query`, `vibes_compile_cache`, `vibes_optimize_budget`, `vibes_verify_gate` |
| **Invarian NFR & Anti-Slop** | Latensi P99 API < 85ms, paritas skema ketat OpenAPI 3.1 / TypeSpec, kontrak error RFC 9457 Problem Details, penegakan Idempotency-Key; kode 100% tuntas tanpa placeholder diverifikasi melalui `node scripts/check-anti-slop.js --strict` |