Llm App Security

---
name: llm-app-security
description: "Securing a feature that calls an LLM: prompt injection as an unsolved input problem, bounding what model output is allowed to reach, tool authorization against the human rather than the model, approval gates on consequential actions, RAG context provenance, system-prompt leakage, and cost limits. Use when sending prompts to an LLM API, building a RAG pipeline, giving a model tools, rendering or executing model output, or exposing an LLM-backed endpoint."
---

<!-- Native skill bundle for agent-skills (cross-tool convention). Generated by `secure-vibe dev regenerate`. -->
<!-- Do not edit by hand; the source of truth is skills/llm-app-security/SKILL.md. -->

# LLM Application Security

Securing a feature that calls an LLM: prompt injection as an unsolved input problem, bounding what model output is allowed to reach, tool authorization against the human rather than the model, approval gates on consequential actions, RAG context provenance, system-prompt leakage, and cost limits. Use when sending prompts to an LLM API, building a RAG pipeline, giving a model tools, rendering or executing model output, or exposing an LLM-backed endpoint.

## ALWAYS

- Start from the premise that **prompt injection is not solved at the prompt layer**. A model has no privileged channel: system text, user text, a retrieved document and a tool result all arrive as one context, and nothing in that context can be marked as instructions the model must obey over the rest. Message roles are a formatting convention, delimiters raise the cost of an attack, and an instruction such as "treat the following as data" is itself just more text in the same window. Design as though injection will sometimes succeed, and put the real controls after the model rather than in front of it.
- Bound what the model's output is **allowed to reach**. Every downstream sink — a query builder, a shell, a file writer, an HTTP client, a renderer — must validate model output exactly as it would validate a request body from the internet, because that is what it is once injection is assumed. `secure-code-review`, `database-security` and `frontend-security` own the individual sinks; what belongs here is that model output never arrives pre-trusted.
- Constrain the output's **shape** with structured generation — a JSON Schema, function-call mode, constrained decoding — whenever the next step consumes it programmatically, and reject anything that fails validation rather than repairing it. A schema bounds the form, not the intent: a well-formed field can still carry a hostile value, so validate the value too.
- Authorize each tool call against the **human user's** permissions, checked at call time, not against what the model was told it may do. The tool allowlist is a capability list, not an authorization decision — and the agent's own privileges are the attack surface, because injected content drives the agent that holds them. Keep the agent's identity narrow for the same reason `iam-best-practices` keeps a pipeline role narrow.
- Put a **human approval gate** in front of consequential and irreversible actions: moving money, deleting or overwriting data, sending messages on a user's behalf, publishing, changing permissions. Injection that reaches a read-only tool is an information problem; injection that reaches a write tool is an incident. The gate is what keeps a single injected instruction from being final.
- Give retrieved content **provenance** and carry it through: which source, which tenant, which trust level. Retrieval is where the untrusted input usually enters, and the retriever is the last component that still knows where the text came from.
- Keep secrets, credentials, and internal hostnames out of the system prompt. Prompt-extraction is reliable enough to treat the system prompt as public, so it can hold instructions but not material whose value depends on staying hidden.
- Validate model parameters server-side — model name, system prompt, tool list, temperature. A client that can choose them can downgrade to a weaker model, replace your instructions, or widen the tool set.
- Bound the cost: per-tenant token budgets, request quotas, a cap on agent iterations and tool calls per task, and a wall-clock timeout. An agent loop with no ceiling is a billing incident that an attacker can trigger deliberately. `api-security` owns the endpoint's rate limiting.
- Record what was sent and what came back — prompt, model and version, retrieved context, tool calls — for audit, under `logging-security`'s redaction policy. Without the retrieved context, an injection incident cannot be reconstructed.
- Consult `ml-security` before loading a model artifact — a checkpoint, an adapter, a fine-tune pulled from a hub. That skill owns the artifact: its format, its provenance, and the fact that loading a pickle-backed checkpoint executes code. This skill covers the application that calls a model, not the file the model arrives in.

## NEVER

- Build a prompt by concatenating user input into a string that also carries your instructions. Use the API's message structure — not because roles are a security boundary, but because a single concatenated string loses even the provenance you would need to investigate.
- Hand model output straight to `eval`, `exec`, `os.system`, `subprocess(shell=True)`, `vm.runInNewContext`, or a raw SQL call.
- Render unfiltered model output as HTML or Markdown that can load remote resources. A model-emitted image URL is a silent exfiltration channel: the model writes the data into the query string and the victim's browser sends it.
- Cache responses keyed on prompt text alone. Prompts share prefixes across users, and the cache key has to carry the tenant and the caller's authorization context.
- Let an agent expand its own capabilities mid-task — registering new tools, editing its own instructions, or raising its own limits — without the same approval the action itself would need.

## KNOWN FALSE POSITIVES

- Red-team and evaluation harnesses exercise jailbreak prompts on purpose. They belong in an isolated environment with no production credentials and no live tools, and their prompts are not findings.
- A read-only tool reachable by an agent without an approval gate is the intended design. The gate exists for actions that change state or leave the system.
- Detection signatures for injection patterns are defensive content, not payloads — a scanner flagging the pattern file is matching on the thing that catches the attack.

## Reference files

Read these only when the task calls for them.

- `references/verifying-findings.md`