What an AI Security Engineer Actually Does

An AI security engineer is the person responsible for making sure that systems built on machine learning models — today, mostly large language models — cannot be turned against the people and organisations that run them. That covers the model, the data it sees, the pipeline that feeds it, and, increasingly, the tools and permissions it holds when it acts as an agent. The job title is new; the discipline underneath it is ordinary security engineering applied to a component that does not behave like ordinary software.

Why the role exists now

Traditional application security assumes code does what it was written to do, so you look for places where untrusted input reaches a dangerous sink. An LLM breaks that assumption. Its instructions and its data travel on the same channel, and it will sometimes follow instructions it finds inside a document, an email or a web page. That single property — explored in depth in prompt injection — means the model itself is a confused deputy waiting to happen.

For a while that was a curiosity, because chatbots could only produce text. Once teams started wiring models into retrieval systems, internal APIs, email, payments and code execution, the consequences became concrete. Someone has to own the question "what happens when the model is wrong on purpose?", and that person is the AI security engineer.

What an AI security engineer protects

The attack surface is wider than the model. In practice, an AI security engineer is responsible for five layers:

The last two rows are where most of the new risk lives. An agent that can read a ticket and issue a refund, or a coding assistant that can run shell commands, turns a text-generation problem into an authorisation problem. Protocols such as the Model Context Protocol make it easy to connect dozens of tools, which is exactly why MCP security and AI agent security have become their own specialisms.

AI security engineer responsibilities, day to day

Threat modelling LLM features

Before anything ships, the AI security engineer maps the system: every input channel (chat, uploads, email ingestion, fetched URLs), every data source the model can retrieve from, and every tool it can call. For each, the question is who can write content that reaches the model, and what the model can do afterwards. The useful output is a short list of trust boundaries and the controls that enforce them — not a fifty-page document.

Red teaming

Red teaming is the hands-on part: authorised, scoped attempts to make the system misbehave. Direct and indirect prompt injection, system prompt extraction, data-boundary tests against retrieval, tool abuse and output-handling attacks. The deliverable is a set of reproducible findings with severity, and — critically — the attacks turned into regression tests. The method is covered in more detail in AI red teaming.

Guardrail design

Guardrails are the layered controls around the model: input normalisation and classification, context separation, output validation, and permission checks at the tool boundary. A good AI security engineer is opinionated about which of these are real boundaries and which are speed bumps. Classifiers and filters reduce attack success rates; authorisation in code and human approval contain the damage when they fail. See LLM guardrails for how the layers fit together.

Secure SDLC for LLM features

• Reviewing prompts, tool schemas and retrieval configuration as security-relevant code.
• Keeping injection and authorisation tests in CI so a one-line prompt edit cannot silently reopen a finding.
• Gating new tools and new data sources behind a lightweight security review.
• Setting rules for model provenance: pinned versions, known publishers, no loading of untrusted serialised model files.

Monitoring and incident response

When something goes wrong, the questions are the usual ones — what happened, to whom, and how do we stop it — but the evidence is different. The AI security engineer makes sure every prompt, retrieval, tool call and output is traced to a real user, that alerts exist for unusual tool sequences and cost spikes, and that there is a tested kill switch for an agent or a single tool. Incident response for an LLM feature usually means revoking a credential, disabling a tool, replaying the trace and fixing the architecture, not just rewording the prompt.

AI security engineer vs AppSec engineer vs ML engineer

The role overlaps with two established ones, and the overlap causes most of the confusion about what it is. The clearest way to see the difference is by what each person is trying to prevent.

In practice, the strongest AI security engineers come from the AppSec side and learn enough about models to reason about their failure modes. That is not a rule — ML engineers with an adversarial mindset make excellent ones too — but most of the real damage in LLM systems still arrives through classic weaknesses: an over-privileged credential, unescaped output rendered as HTML, a missing authorisation check. The model is the new entry point; the blast radius is usually old-fashioned.

The frameworks an AI security engineer uses

OWASP Top 10 for LLM Applications (2025)

The OWASP Top 10 for LLM Applications, maintained by the OWASP GenAI Security Project, is the most practical checklist for application-level work. The 2025 edition lists:

1. LLM01 Prompt Injection
2. LLM02 Sensitive Information Disclosure
3. LLM03 Supply Chain
4. LLM04 Data and Model Poisoning
5. LLM05 Improper Output Handling
6. LLM06 Excessive Agency
7. LLM07 System Prompt Leakage
8. LLM08 Vector and Embedding Weaknesses
9. LLM09 Misinformation
10. LLM10 Unbounded Consumption

I use it as a coverage check at the end of a threat model rather than as the starting point; a walkthrough of each entry with concrete tests is in OWASP Top 10 for LLM applications.

MITRE ATLAS

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a knowledge base of adversary tactics and techniques against AI-enabled systems, modelled on MITRE ATT&CK and backed by real-world case studies. It is most useful when you need a shared vocabulary with a security operations or threat intelligence team, or when you are planning a red team engagement and want to make sure you have considered the full attack lifecycle rather than only the prompt.

NIST AI Risk Management Framework

The NIST AI RMF is a voluntary framework organised around four functions — Govern, Map, Measure and Manage — with a Generative AI Profile (NIST AI 600-1) published in 2024. It sits at the governance level: it will not tell you how to stop an injection, but it is the framework leadership, auditors and procurement teams tend to ask about. An AI security engineer who can map technical controls onto it is far more effective in a larger organisation.

AI security engineer skills

• Web and API security fundamentals. Authentication, authorisation, injection, SSRF, XSS. Most LLM exploits end in one of these.
• Hands-on LLM building. You cannot secure a RAG pipeline or an agent loop you have never built. Embeddings, vector databases, tool calling and orchestration should feel familiar.
• Adversarial creativity. The ability to think in terms of what content can reach the model and who wrote it — and to keep iterating when the first ten payloads fail.
• Secure architecture. Least privilege, user-scoped credentials, separation of trusted and untrusted context, human-in-the-loop design.
• Testing automation. Turning manual findings into repeatable test suites that run in CI.
• Clear writing. Findings that engineers can reproduce and fix, and summaries that leadership can act on.

If you are working out how to build these skills, I have written a staged roadmap in how to become an AI security engineer.

How I work as an AI security engineer

My own work sits across both sides of the line: I build AI systems — RAG chatbots on FastAPI and Qdrant, agent systems, and an "AI Employee" with nine orchestrated services — and I test them and others' systems for weaknesses. As an AI security engineer, I (Sheikh Mujtaba) have found that building first makes the testing far sharper, because I know where the shortcuts usually get taken.

A typical engagement follows the same shape:

1. Scope in writing. What is in bounds, which environment, which accounts, and what counts as a finding. AI red teaming is an authorised security exercise like any other.
2. Map the system as built, not as described. I list every input channel, every retrieval source and every tool, and I check which credentials each tool actually uses. This step alone usually surfaces the most serious issue — an agent calling an internal API with a service account that can see everything.
3. Classic web testing first. Authentication, access control and output handling on the surrounding application. My background in web application pentesting, malware analysis and WAF bypass research matters here more than any prompt trick.
4. LLM-specific testing. Direct and indirect injection through every channel, system prompt extraction, retrieval boundary tests, tool-abuse chains and cost-exhaustion inputs.
5. Fix the architecture, not the wording. My recommendations focus on permission checks in tools, user-scoped credentials, strict context grounding for retrieval, and human approval for irreversible actions — the same human-in-the-loop pattern I use in my own agent builds.
6. Leave regression tests behind. Every confirmed finding becomes a test so the next prompt change cannot quietly undo the fix.

Practising in public red-teaming programmes, such as the Gray Swan arena engagements I have taken part in, keeps the attacker side of my thinking current; models change quickly, and techniques that failed six months ago sometimes work again after an update. If you need this kind of assessment or want security built into an AI feature from the start, my services page covers how I work with teams.

Where the role is heading

The centre of gravity is moving from chatbots to agents. As models gain the ability to browse, write code, send messages and call business systems, the AI security engineer's work looks less like content moderation and more like identity and access management for a non-human actor that can be socially engineered. Expect more emphasis on tool permission design, agent-to-agent trust, and audit trails — and less on clever single-prompt jailbreaks. For the broader control stack, start with the AI security guide.

Key takeaways

• An AI security engineer secures systems where a model is in the decision path: models, data, pipelines, agents and tool integrations.
• The defining problem is that a model cannot reliably separate instructions from data, so controls must live outside the model.
• Day-to-day work is threat modelling, red teaming, guardrail design, secure SDLC for LLM features and incident response.
• The role differs from AppSec in its mental model of the component and from ML engineering in its goal — preventing abuse, not improving accuracy.
• OWASP Top 10 for LLM Applications, MITRE ATLAS and the NIST AI RMF give coverage, adversary vocabulary and governance language respectively.
• Strong web security fundamentals plus hands-on LLM building experience is the most reliable skill combination.