AI Red Teaming: A Practical Methodology

AI red teaming is the practice of attacking an AI system the way a real adversary would, in order to find failures before someone else does. For LLM applications that means more than trying jailbreak prompts against a chatbot. The model is one component; the real targets are the tools it can call, the data it can reach and the systems that trust its output. This guide is the methodology I use, phase by phase, from scoping through to regression tests in CI.

The methodology at a glance

Phase 1: Scoping and rules of engagement

AI red teaming is an authorised security exercise like any penetration test. Before sending a single prompt, get the following agreed in writing:

• Targets: which applications, endpoints, models and environments. Prefer staging with production-like data shapes but no real customer data.
• Objectives: what counts as a win. "Exfiltrate another user's data" and "trigger an unapproved payment" are better objectives than "make it say bad things".
• Out of bounds: third-party providers' infrastructure, real customers, volume that could breach the provider's terms or run up an unbounded bill.
• Accounts and data: dedicated test users at each privilege level, plus seeded canary records so data leaks are easy to detect.
• Stop conditions and contacts: who to call if you find something critical, and when testing must pause.
• Harm-content handling: if the scope includes harmful-content testing, agree how transcripts are stored and who may see them.

Phase 2: Threat model the system

Draw the data flow before you attack it. For each component, ask who can write to it, what the model can do with it, and what trusts the model's output. The OWASP Top 10 for LLM applications is the coverage checklist I map against at this stage.

1. Entry points: chat, API, file uploads, email or ticket ingestion, web pages the system fetches, webhooks, memory.
2. Data sources: vector stores, databases, document shares, and whose permissions are used to read them.
3. Tools and actions: every function the model can call, the credentials behind it, and whether any action runs without a human.
4. Output sinks: where responses are rendered or executed: browsers, SQL, shells, emails, other agents.
5. Trust boundaries: every point where untrusted content can reach the context window of a call that holds privileged tools.

The output is a short list of attack goals ranked by impact. An agent with a refund tool and an inbox it reads automatically will usually top the list, because it combines untrusted input with real-world effect.

Phase 3: Build the attack corpus

A corpus is a versioned set of attack cases, each tagged with technique, target goal and the OWASP risk it exercises. Organise it by technique rather than by individual prompt, because single prompts go stale as models change while techniques persist.

Add the canary values from Phase 1 to your detection logic: if a canary string from another test tenant ever appears in a response, you have a data-boundary finding without needing to judge the text.

Phase 4: Manual testing

Manual testing finds the failures that matter most: the ones specific to this application's tools, data and business logic. A scanner does not know that an order ID in a support ticket is enough to trigger a refund. A human reading the tool list does.

In my own red-teaming work (Sheikh Mujtaba), including Gray Swan ethical red-teaming engagements, the pattern I see most often is that persistence and adaptation beat any single clever prompt. You read the refusal, work out which part of the request triggered it, change one variable and try again. Keep notes on every attempt, including failures, because the path to a working attack is part of the finding.

• Establish a baseline first: what does correct behaviour look like for each feature?
• Change one variable at a time so you know what actually caused a bypass.
• Re-run every success several times; LLM outputs are non-deterministic, and a one-in-twenty success is still a finding.
• Chain small weaknesses. A leaked system prompt reveals tool names; tool names make injection payloads precise.

Phase 5: Automated testing and AI red teaming tools

Automation gives breadth: hundreds of variants, every technique family, every release. Three open-source tools are worth knowing, and they suit different jobs.

Two cautions. First, automated scorers make mistakes in both directions, so sample and manually review results, especially anything scored as a pass on a high-risk goal. Second, point scanners at your application endpoint, not only at the bare model. The bare model has no tools, no retrieval and no output sinks, so it cannot show you most of the risks that matter.

Phase 6: Testing agents and tools

Agents raise the stakes because a successful injection becomes an action. The question shifts from "what will it say?" to "what will it do, and with whose permissions?" I cover the defensive side in AI agent security and MCP security; the offensive checklist is:

1. Unauthorised tool reach: can a low-privilege user cause a tool call that should require higher privilege?
2. Indirect injection to action: plant instructions in content the agent reads (a ticket, a fetched page, a calendar invite) and see whether any tool fires.
3. Tool chaining: combine two allowed actions into a forbidden effect, such as reading a private record and then emailing it externally.
4. Parameter tampering: coax the model into passing another user's ID, an oversized amount or a path traversal string to a tool.
5. Approval bypass: check that human-approval gates cannot be skipped by rephrasing, splitting an action or retrying.
6. Runaway loops: trigger failing tool calls and confirm step limits and budgets actually stop the agent.

Assert on tool calls, not on text. The model may refuse politely in its reply while having already called the tool in the same turn, so instrument the tool layer and treat any unexpected call as the finding.

Phase 7: Measuring and reporting

Because LLM behaviour is probabilistic, a finding needs a rate as well as a reproduction. Run each successful attack a fixed number of times (for example, twenty) and report the attack success rate: successful attempts divided by total attempts. Report the model version, temperature and system prompt version alongside it, because any of these can change the number.

• Severity comes from impact and reachability: who can trigger it, what they gain, and whether user interaction is needed. A low success rate does not make a critical impact low severity; attackers can retry.
• Reproducibility comes from complete transcripts, exact payloads, environment details and the success rate.
• Mapping to OWASP LLM IDs gives developers and auditors a shared reference.
• Remediation should name an architectural control, not just a prompt edit. Prompt edits are the weakest fix and the easiest to regress.

## Finding: Indirect prompt injection triggers refund tool

**ID:** AIRT-007
**Severity:** High
**OWASP mapping:** LLM01:2025 Prompt Injection, LLM06:2025 Excessive Agency
**Component:** Support triage agent, issue_refund tool
**Environment:** staging, model <name/version>, temperature <x>, system prompt v<n>

### Summary
Instructions placed in a customer support ticket cause the triage agent
to call issue_refund for an order the ticket author does not own.

### Preconditions
- Attacker can submit a support ticket (any unauthenticated user).
- Agent processes new tickets automatically.

### Steps to reproduce
1. Submit a ticket whose body contains the payload in Appendix A.
2. Wait for the triage agent to process the ticket.
3. Observe the tool-call log for issue_refund with order <canary order ID>.

### Attack success rate
<successes>/<attempts> runs on <date>.

### Impact
Unauthenticated users can trigger refunds against arbitrary orders up to
the auto-approve ceiling.

### Root cause
issue_refund authorises against the agent's service account, not the
ticket author's identity. Untrusted ticket text shares a context with the
privileged tool.

### Recommended fix
- Authorise refunds against the ticket author's identity inside the tool.
- Split summarisation of untrusted tickets into a tool-less call.
- Require human approval for all refunds initiated from ticket content.

### Retest
Add the Appendix A payload and its variants to the CI regression suite.

### Appendix A: payload and full transcript
<redacted payload and transcript>

Phase 8: Regression testing in CI

Every confirmed finding becomes a test. LLM applications regress silently: a one-line prompt change, a new tool or a model upgrade can reopen something that was fixed months ago. Run the suite on every change to prompts, tools, retrieval configuration or model version.

# Example CI job: run the red-team regression suite on relevant changes.
name: llm-red-team-regression
on:
  pull_request:
    paths:
      - "prompts/**"
      - "agents/tools/**"
      - "config/model.yaml"
jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements-dev.txt
      # Each case replays a confirmed finding N times against staging
      # and fails if any privileged tool fires or a canary value leaks.
      - run: pytest tests/redteam --runs-per-case 10

Gate on tool calls and canary leaks, which are deterministic to detect, rather than on judging response text. Track attack success rate per case over time so you notice a fix weakening before it fails outright.

Lessons from competitive red-teaming arenas

Competitive arenas such as Gray Swan's, where many researchers attack the same models against defined objectives, teach habits that transfer directly to client work:

• Objectives must be precise. A clear success condition, such as a specific tool call with specific arguments, prevents arguments about whether a borderline output counts. Write your own scope this way.
• Defences that look strong in isolation often fall to combined techniques. Framing plus obfuscation plus multi-turn setup is the norm, not the exception.
• Indirect injection through tool output is a rich attack surface. Content the agent fetches is often treated with less suspicion than the user's own message.
• Refusals are signal. How a model refuses tells you which part of the request it reacted to.
• Document as you go. A success you cannot reproduce is not a finding.

Key takeaways

• AI red teaming targets the whole system: tools, data, output sinks and permissions, not just the model's words.
• Start with written scope, test accounts and canary data, then threat model before attacking.
• Organise the attack corpus by technique and map it to the OWASP LLM Top 10.
• Use manual testing for depth and tools such as PyRIT, garak and promptfoo for breadth.
• For agents, assert on tool calls and canary leaks rather than response text.
• Report severity, reproduction steps and attack success rate, and turn every finding into a CI regression test.