Google Cloud Solution Multi Agent Security

---
name: google-cloud-solution-multi-agent-security
metadata:
  category: MultiProductSolutions
description: >-
  Designs, deploys, and secures Google Cloud Agent Gateway solutions.
  Use when the user needs to configure multi-agent security, ingress (CLIENT_TO_AGENT), or egress (AGENT_TO_ANYWHERE) patterns involving Model Armor, IAP, and Agent Registry.
  Don't use for general Cloud Load Balancing or basic VPC setup not related to Agent Gateways.
---

# Agent Gateway multi-agent security

## Critical Enforcement Rules & Rationale

*   **Gcloud Release Tracks**: Always use the exact release tracks specified in
    the commands (e.g., `gcloud beta network-services agent-gateways`). Omitting
    these prefixes causes commands to fail because Agent Gateway features are
    located in specialized, non-default namespaces.
*   **API Enablement**: Include `modelarmor.googleapis.com` in the API
    enablement list when setting up guardrails. Excluding it prevents Model
    Armor policies and filters from successfully attaching to the Gateway.
*   **Egress Verification**: Egress policy verification requires using the
    Python script
    ([scripts/verify_egress_policies.py](scripts/verify_egress_policies.py)),
    not `curl`. Egress gateways rely on runtime SDK lifecycle handling and JWT
    context that a standard curl command cannot simulate correctly.
*   **Model Armor Keys**: In `model-armor-config.yaml`, always include both
    `piAndJailbreakFilterSettings` and `sdpFilterSettings` (`filterEnforcement:
    ENFORCE`). Invalid or missing filters cause deployment validation failures
    or lead to silent bypasses of the guardrails.
*   **Subnet Private Access**: Any subnet hosting a Private Service Connect
    network attachment for Egress Gateways must have `private_ip_google_access =
    true` enabled in Terraform. Disabling this blocks connectivity to
    Google-managed endpoints, causing total routing failures for agents.
*   **Direct Delivery**: Immediately provide the requested architecture,
    configuration files, CLI commands, scripts, and diagrams in full. Do not
    stop at a planning phase, do not generate a plan artifact, and do not ask
    for user confirmation before delivering outputs.
*   **No Infrastructure Execution**: Do not attempt to run deployment or
    verification commands (such as `gcloud`, `kubectl`, `terraform`, or `curl`)
    against real cloud resources during design. You are generating plan
    configurations, not executing them.

> [!IMPORTANT] **Just-In-Time (JIT) Resource Loading Protocol:** Inspect
> template files in [assets/](assets/) and executable scripts in
> [scripts/](scripts/) using `view_file` as needed for extended configurations,
> deployment scripts, and test suites.

--------------------------------------------------------------------------------

## Quick Reference: Required Filenames

Always generate files with these exact names when requested:

1.  `agw-ingress-config.yaml`
    ([assets/agw-ingress-config.yaml](assets/agw-ingress-config.yaml))
2.  `agw-egress-config.yaml`
    ([assets/agw-egress-config.yaml](assets/agw-egress-config.yaml))
3.  `agw-authz-extension.yaml`
    ([assets/agw-authz-extension.yaml](assets/agw-authz-extension.yaml))
4.  `agw-authz-policy.yaml`
    ([assets/agw-authz-policy.yaml](assets/agw-authz-policy.yaml))
5.  `model-armor-config.yaml`
    ([assets/model-armor-config.yaml](assets/model-armor-config.yaml))
6.  `sgp-policy.yaml` ([assets/sgp-policy.yaml](assets/sgp-policy.yaml))
7.  `iap-policy.json` ([assets/iap-policy.json](assets/iap-policy.json))
8.  `model-armor-payload.json`
    ([assets/model-armor-payload.json](assets/model-armor-payload.json))

--------------------------------------------------------------------------------

## 1. Dual Ingress & Egress Architecture Design (`dual_ingress_egress_architecture_design`)

-   **Ingress Pattern**: `CLIENT_TO_AGENT` fronted by Ingress Control Plane
    (Agent Gateway, Model Armor).
-   **Egress Pattern**: `AGENT_TO_ANYWHERE` utilizing Egress Control Plane
    (Agent Gateway, `roles/iap.egressor` CEL policies, Cloud DNS) and Egress
    Data Plane (PSC Interface, Cloud Run, PSC Google APIs Global Endpoint),
    coordinated via Agent Registry & Agent Engine runtime.
-   **Mermaid Diagram**:

    ```mermaid
    graph TD
        Client["External Clients"] -->|HTTPS / MCP| GLB["Global Load Balancer"]
        GLB --> Ingress["Ingress Agent Gateway (CLIENT_TO_AGENT)"]
        Ingress --> MA["Model Armor (CONTENT_AUTHZ)"]
        MA --> Agent["Agent Engine Agents (BillingAgent, SupportAgent, FraudAgent)"]
        Agent --> Egress["Egress Agent Gateway (AGENT_TO_ANYWHERE)"]
        Egress --> PSC["Private Service Connect Network Attachment"]
        PSC --> Tools["Private MCP Tool Backends"]
    ```

--------------------------------------------------------------------------------

## 2. Ingress & Egress Guardrail Policy Config (`ingress_and_egress_guardrail_policy_config`)

When requested for Ingress & Egress guardrail policy configs, you MUST generate
and create all required files in the workspace:

-   `agw-ingress-config.yaml`
    ([assets/agw-ingress-config.yaml](assets/agw-ingress-config.yaml)): Declares
    `governedAccessPath: CLIENT_TO_AGENT` with protocols `HTTP` and `MCP`.
-   `agw-egress-config.yaml`
    ([assets/agw-egress-config.yaml](assets/agw-egress-config.yaml)): Declares
    `governedAccessPath: AGENT_TO_ANYWHERE` with protocol `MCP`.
-   `agw-authz-extension.yaml`
    ([assets/agw-authz-extension.yaml](assets/agw-authz-extension.yaml)):
    Configures AuthzExtension service for IAP authorization.
-   `agw-authz-policy.yaml`
    ([assets/agw-authz-policy.yaml](assets/agw-authz-policy.yaml)): Configures
    `AuthzPolicy` action `ALLOW` targeting both Ingress and Egress gateways.
-   `iap-policy.json` ([assets/iap-policy.json](assets/iap-policy.json)): Binds
    `roles/iap.egressor` with CEL condition checking
    `iap.googleapis.com/mcp.toolName == 'get_account_balance' &&
    iap.googleapis.com/mcp.tool.isReadOnly == true`.
-   `model-armor-config.yaml`
    ([assets/model-armor-config.yaml](assets/model-armor-config.yaml)): Enables
    `piAndJailbreakFilterSettings` and `sdpFilterSettings` with
    `filterEnforcement: ENFORCE`.
-   `sgp-policy.yaml` ([assets/sgp-policy.yaml](assets/sgp-policy.yaml)):
    Implements Natural Language Constraints blocking transactions > $1000 and
    sanitizing PII.

--------------------------------------------------------------------------------

## 3. Ingress & Egress Infrastructure Deployment (`ingress_and_egress_infrastructure_deployment`)

Inspect and provide the step-by-step `gcloud` CLI commands from
[scripts/deploy_infrastructure.sh](scripts/deploy_infrastructure.sh):

1.  **Enable Required APIs**: `compute`, `networkservices`, `networksecurity`,
    `modelarmor`, `iap`, `agentregistry`, `serviceextensions`, and `aiplatform`.
2.  **Import Agent Gateways**: Ingress (`agw-ingress-config.yaml`) and Egress
    (`agw-egress-config.yaml`) via `gcloud alpha network-services agent-gateways
    import`.
3.  **Import Authz Extension**: `agw-authz-extension.yaml` via `gcloud beta
    service-extensions authz-extensions import`.
4.  **Import Authz Policy**: `agw-authz-policy.yaml` via `gcloud beta
    network-security authz-policies import`.

--------------------------------------------------------------------------------

## 4. Ingress & Egress Security Validation (`ingress_and_egress_security_validation`)

When validating security for Ingress and Egress:

1.  **Ingress 403 Unauthenticated Test**: Provide the copy-pasteable
    verification curl command from
    [scripts/validate_ingress_unauth.sh](scripts/validate_ingress_unauth.sh)
    sending an unauthenticated POST request to the Reasoning Engine endpoint
    expecting HTTP 403 Forbidden.
2.  **Python Egress Verification Script (MUST use Python script snippet, NOT
    curl)**: Provide the Python verification script snippet from
    [scripts/verify_egress_policies.py](scripts/verify_egress_policies.py)
    sending JSON-RPC `tools/call` requests (`get_account_balance`) through the
    Egress Gateway to verify HTTP 200 for allowed tools.
3.  **Model Armor Test Payload**: Generate `model-armor-payload.json`
    ([assets/model-armor-payload.json](assets/model-armor-payload.json))
    containing prompt injection/jailbreak instructions.

--------------------------------------------------------------------------------

## 5. Troubleshooting Ingress & Egress Failures (`troubleshooting_ingress_and_egress_failures`)

-   **Ingress 403 (Client-to-Agent)**:

    -   **Root Cause**: Unauthenticated client requests or missing/invalid OAuth
        2.0 / IAP identity tokens.
    -   **OAuth Configuration Steps**:
        1.  Configure OAuth 2.0 Client ID credentials in Google Cloud Console.
        2.  Grant the client identity / service account
            `roles/iap.httpsResourceAccessor` permission.
        3.  Exchange credentials with Google OAuth to acquire an OIDC / OAuth ID
            token.
        4.  Pass the token in the `Authorization: Bearer <TOKEN>` header.
    -   **Verification Command**: Provide the curl command from
        [scripts/verify_ingress_auth.sh](scripts/verify_ingress_auth.sh).

-   **Egress 403 (Agent-to-Anywhere)**:

    -   **Root Cause**: Missing `roles/iap.egressor` IAM bindings on the Agent
        Identity, malformed principal ID, or mismatched CEL condition on tool
        metadata.
    -   **Fix Command**: Provide the exact `gcloud` command from
        [scripts/fix_egress_iap.sh](scripts/fix_egress_iap.sh).

--------------------------------------------------------------------------------

## 6. Hybrid VPN Connectivity & Egress Routing (`hybrid_vpn_connectivity_egress_routing`)

-   **Terraform HCL**: Refer to baseline Terraform config in
    [assets/main.tf](assets/main.tf) for VPC, subnets
    (`private_ip_google_access = true`), PSC network attachment, Cloud DNS
    private forwarding for `aws.internal.`, and HA VPN gateway/router.
-   **Egress Gateway Config (`agw-egress-config.yaml`)**: Generate configuration
    declaring `governedAccessPath: AGENT_TO_ANYWHERE`, pointing to the PSC
    network attachment, and referencing `aws.internal.` in `dnsPeeringConfig`
    (see [assets/agw-egress-config.yaml](assets/agw-egress-config.yaml)).
-   **Python SDK Deployment Script**: Refer to
    [scripts/hybrid_vpn_agent.py](scripts/hybrid_vpn_agent.py) for the complete
    script initializing Vertex AI with `agent_to_anywhere_config` referencing
    the Egress Gateway, enabling telemetry, and deploying `HybridAgent` using
    `types.IdentityType.AGENT_IDENTITY`.

--------------------------------------------------------------------------------

## 7. Private Egress GKE Internal Load Balancer (`private_egress_gke_internal_load_balancer`)

-   Expose GKE internal MCP tool server via an Internal Load Balancer (ILB) at
    literal IP `10.0.1.50`, connecting via Agent Gateway PSC Interface + Cloud
    DNS Private zone.
-   **Cloud DNS Record Mapping**: Provide the command from
    [scripts/create_gke_dns_record.sh](scripts/create_gke_dns_record.sh) mapping
    the private domain to GKE's private ILB IP `10.0.1.50`.
-   **Explicit TLS Warning**: Agent Gateway egress **does not natively trust
    self-signed certificates or private enterprise CAs**. You **must** use
    publicly trusted TLS certificates signed by a trusted Certificate Authority
    (e.g., Let's Encrypt).

--------------------------------------------------------------------------------

## 8. Governance Controls Model Armor SGP (`governance_controls_model_armor_sgp`)

When configuring dual safety layers with Model Armor on Ingress and SGP on
Egress:

1.  **Model Armor Config**: Generate `model-armor-config.yaml`
    ([assets/model-armor-config.yaml](assets/model-armor-config.yaml)) with
    `piAndJailbreakFilterSettings` and `sdpFilterSettings` (`filterEnforcement:
    ENFORCE`).
2.  **Semantic Governance Policy**: Generate `sgp-policy.yaml`
    ([assets/sgp-policy.yaml](assets/sgp-policy.yaml)) with Natural Language
    Constraints blocking transactions > $1000 and sanitizing PII.
3.  **Curl PATCH Command**: Provide the curl command from
    [scripts/enforce_sgp_patch.sh](scripts/enforce_sgp_patch.sh) to update
    `authzExtensions` with `sgpEnforcementMode` set to `ENFORCE`.

--------------------------------------------------------------------------------

## 9. Multi-Agent Cloud Run Egress Routing (`multi_agent_cloud_run_egress_routing`)

Do NOT produce a plan artifact or stop at planning. When configuring multi-agent
Cloud Run egress routing, you MUST directly provide and generate ALL required
components:

1.  **Egress Gateway Config (`agw-egress-config-run.yaml`)**: Generate
    configuration declaring `governedAccessPath: AGENT_TO_ANYWHERE`, PSC network
    attachment, and DNS peering for `*.run.app` (see
    [assets/agw-egress-config-run.yaml](assets/agw-egress-config-run.yaml)).
2.  **Register Cloud Run Services in Agent Registry**: Provide the registration
    commands from
    [scripts/register_cloud_run_services.sh](scripts/register_cloud_run_services.sh)
    registering all 3 Cloud Run services (`marketing-tool-service`,
    `sales-tool-service`, `support-tool-service`) in the `us-east4` Agent
    Registry.
3.  **`iap-policy.json` (Multi-Agent)**: Generate `iap-policy.json`
    ([assets/iap-policy-multi-agent.json](assets/iap-policy-multi-agent.json))
    containing all 3 `principal://` bindings in the `members` list under
    `roles/iap.egressor`.
4.  **Python SDK Deployment Script**: Refer to
    [scripts/multi_agent_cloud_run.py](scripts/multi_agent_cloud_run.py) for the
    complete GenAI SDK deployment script.

--------------------------------------------------------------------------------

## 10. Advanced Model Armor Filtering (`advanced_model_armor_filtering`)

For custom keyword matching, configure `userDefinedFilterSettings` (see
[assets/model-armor-advanced.yaml](assets/model-armor-advanced.yaml)).

--------------------------------------------------------------------------------

## 11. Known Traps & Gotchas (`known_traps_and_gotchas`)

*   **`network_attachment` is `ForceNew`**: Enabling Semantic Governance
    Policies (SGP) or modifying network attachments after the initial Terraform
    apply will force-recreate the gateway resource. If not managed carefully,
    this can cause dependency deadlocks during destroy operations. Plan
    infrastructure sequencing accordingly.
*   **Authz Policy Limit**: An Agent Gateway allows at most **4 custom
    authorization policies** attached concurrently. Ensure your security posture
    consolidates rules within this limit.