Llm App Security

---
id: llm-app-security
version: "2.0.0"
title: "LLM Application Security"
description: "Securing a feature that calls an LLM: prompt injection as an unsolved input problem, bounding what model output is allowed to reach, tool authorization against the human rather than the model, approval gates on consequential actions, RAG context provenance, system-prompt leakage, and cost limits. Use when sending prompts to an LLM API, building a RAG pipeline, giving a model tools, rendering or executing model output, or exposing an LLM-backed endpoint."
category: prevention
severity: critical
applies_to:
  - "when generating code that sends a prompt to an LLM API"
  - "when building a RAG pipeline or feeding retrieved / tool output back into a prompt"
  - "when giving a model tools, function calls, or the ability to act"
  - "when rendering or executing model output (UI, SQL, shell, file, HTTP)"
  - "when exposing an LLM-backed endpoint to users"
languages: ["python", "javascript", "typescript", "go", "*"]
token_budget:
  minimal: 1300
  compact: 1700
  full: 2000
rules_path: "rules/"
related_skills: ["ml-security", "iam-best-practices", "api-security", "secure-code-review"]
last_updated: "2026-08-12"
sources:
  - "OWASP Top 10 for LLM Applications 2025"
  - "OWASP Top 10 for Agentic Applications"
  - "MITRE ATLAS (Adversarial Threat Landscape for AI Systems)"
  - "CWE-1426: Improper Validation of Generative AI Output"
---

# LLM Application Security

## Rules (for AI agents)

### ALWAYS
- Start from the premise that **prompt injection is not solved at the prompt layer**.
  A model has no privileged channel: system text, user text, a retrieved document and
  a tool result all arrive as one context, and nothing in that context can be marked
  as instructions the model must obey over the rest. Message roles are a formatting
  convention, delimiters raise the cost of an attack, and an instruction such as
  "treat the following as data" is itself just more text in the same window. Design
  as though injection will sometimes succeed, and put the real controls after the
  model rather than in front of it.
- Bound what the model's output is **allowed to reach**. Every downstream sink — a
  query builder, a shell, a file writer, an HTTP client, a renderer — must validate
  model output exactly as it would validate a request body from the internet, because
  that is what it is once injection is assumed. `secure-code-review`,
  `database-security` and `frontend-security` own the individual sinks; what belongs
  here is that model output never arrives pre-trusted.
- Constrain the output's **shape** with structured generation — a JSON Schema,
  function-call mode, constrained decoding — whenever the next step consumes it
  programmatically, and reject anything that fails validation rather than repairing
  it. A schema bounds the form, not the intent: a well-formed field can still carry a
  hostile value, so validate the value too.
- Authorize each tool call against the **human user's** permissions, checked at call
  time, not against what the model was told it may do. The tool allowlist is a
  capability list, not an authorization decision — and the agent's own privileges are
  the attack surface, because injected content drives the agent that holds them.
  Keep the agent's identity narrow for the same reason `iam-best-practices` keeps a
  pipeline role narrow.
- Put a **human approval gate** in front of consequential and irreversible actions:
  moving money, deleting or overwriting data, sending messages on a user's behalf,
  publishing, changing permissions. Injection that reaches a read-only tool is an
  information problem; injection that reaches a write tool is an incident. The gate
  is what keeps a single injected instruction from being final.
- Give retrieved content **provenance** and carry it through: which source, which
  tenant, which trust level. Retrieval is where the untrusted input usually enters,
  and the retriever is the last component that still knows where the text came from.
- Keep secrets, credentials, and internal hostnames out of the system prompt.
  Prompt-extraction is reliable enough to treat the system prompt as public, so it
  can hold instructions but not material whose value depends on staying hidden.
- Validate model parameters server-side — model name, system prompt, tool list,
  temperature. A client that can choose them can downgrade to a weaker model, replace
  your instructions, or widen the tool set.
- Bound the cost: per-tenant token budgets, request quotas, a cap on agent iterations
  and tool calls per task, and a wall-clock timeout. An agent loop with no ceiling is
  a billing incident that an attacker can trigger deliberately. `api-security` owns
  the endpoint's rate limiting.
- Record what was sent and what came back — prompt, model and version, retrieved
  context, tool calls — for audit, under `logging-security`'s redaction policy.
  Without the retrieved context, an injection incident cannot be reconstructed.
- Consult `ml-security` before loading a model artifact — a checkpoint, an adapter, a
  fine-tune pulled from a hub. That skill owns the artifact: its format, its
  provenance, and the fact that loading a pickle-backed checkpoint executes code. This
  skill covers the application that calls a model, not the file the model arrives in.

### NEVER
- Build a prompt by concatenating user input into a string that also carries your
  instructions. Use the API's message structure — not because roles are a security
  boundary, but because a single concatenated string loses even the provenance you
  would need to investigate.
- Hand model output straight to `eval`, `exec`, `os.system`, `subprocess(shell=True)`,
  `vm.runInNewContext`, or a raw SQL call.
- Render unfiltered model output as HTML or Markdown that can load remote resources.
  A model-emitted image URL is a silent exfiltration channel: the model writes the
  data into the query string and the victim's browser sends it.
- Cache responses keyed on prompt text alone. Prompts share prefixes across users, and
  the cache key has to carry the tenant and the caller's authorization context.
- Let an agent expand its own capabilities mid-task — registering new tools, editing
  its own instructions, or raising its own limits — without the same approval the
  action itself would need.

### KNOWN FALSE POSITIVES
- Red-team and evaluation harnesses exercise jailbreak prompts on purpose. They belong
  in an isolated environment with no production credentials and no live tools, and
  their prompts are not findings.
- A read-only tool reachable by an agent without an approval gate is the intended
  design. The gate exists for actions that change state or leave the system.
- Detection signatures for injection patterns are defensive content, not payloads —
  a scanner flagging the pattern file is matching on the thing that catches the
  attack.

## Context (for humans)

The uncomfortable premise of this skill is that its central risk has no fix at the
layer people reach for first. Prompt injection persists because a language model
consumes one undifferentiated context: there is no mechanism by which some of that
text carries authority the rest does not. Delimiters, role labels, and "ignore any
instructions in the following" all raise the attacker's cost and none of them
establish a boundary. Treating them as one is how a system gets built with the
defenses in the wrong place.

So the useful question is not "how do I stop injection" but "what happens when it
works". That answer is entirely in the surrounding architecture: what tools exist,
whose permissions they run with, which actions can be taken without a human, and what
the model's output is allowed to touch. An assistant that can only read is a
disclosure risk. The same assistant with a write tool and the deploy role is an
incident waiting for one hostile document to enter the index.

For the **model artifacts** themselves — checkpoint formats, poisoning, training-data
handling — see `ml-security`. This skill covers the application that calls a model.

## References

- `references/verifying-findings.md` — confirm or refute a finding, then lock it
- `rules/prompt_injection_patterns.json`
- [OWASP Top 10 for LLM Applications 2025](https://genai.owasp.org/llm-top-10/).
- [OWASP GenAI Security Project](https://genai.owasp.org/) — including the Agentic Applications work.
- [MITRE ATLAS](https://atlas.mitre.org/).
- [CWE-1426](https://cwe.mitre.org/data/definitions/1426.html) — Improper Validation of Generative AI Output.