Course Outline
Module 1: How AI Apps Break
- LLM, RAG, and agent architectures from the developer's side
- The request and response lifecycle of an AI feature
- Prompt flow: system, developer, user, and tool messages
- Where untrusted data enters and re-enters the model
- The trust boundaries a developer owns versus inherits
- Why AI attacks are semantic, not syntactic
- Mapping the OWASP LLM Top 10 to code you write
Module 2: Prompt Injection for Builders
- Direct versus indirect prompt injection
- Hidden instructions in documents, web pages, and tool output
- Jailbreaks and role confusion
- Why instruction and data separation matters
- Defensive prompt design: delimiters, structure, minimal authority
- Why prevention is partial, and designing for containment
- Lab 01: attack your own chatbot, bypass a naïve filter, restructure the prompt
Module 3: Treating Model Output as Untrusted
- Model output as untrusted input to the rest of the application
- Insecure output handling: XSS, SSRF, and downstream command or SQL injection
- Never eval, exec, or render raw model output
- Structured outputs and schema validation
- Output encoding and allowlists
- Safe rendering in web and UI contexts
- Lab 02: find and fix an insecure-output-handling bug
Module 4: RAG Security
- Vector database and retrieval threats
- Ingestion sanitization
- Document provenance and trust scoring
- Retrieval scoping and metadata isolation
- Hidden instructions in retrieved content
- Data exfiltration through retrieval
- Lab 03: poison a RAG pipeline, then defend it
Module 5: Agent and Tool Safety
- Excessive agency and tool abuse
- Least privilege for agents
- Tool allowlists and argument validation
- Approval gates and human-in-the-loop
- Sandboxing tool execution
- Scoped, short-lived credentials and limiting autonomous loops
- Lab 04: lock down an over-permissioned agent
Module 6: Secrets, Identity, and Cost
- API key and secret management: never in prompts, code, or logs
- Per-user authentication and authorization for AI features
- Propagating user identity to tools and retrieval
- Denial of wallet: unbounded token and cost consumption
- Rate limits, token budgets, and timeouts
- Logging without leaking secrets or PII
- Lab 05: get secrets out of the prompt path and add cost limits
Module 7: Guardrail Libraries
- What guardrail frameworks do and do not do
- Input guardrails: injection, PII, and topic classifiers
- Output guardrails: validation, filtering, grounding checks
- When a guardrail is right versus your own deterministic check
- Layering guardrails with the controls from earlier modules
- Performance, false positives, and failure modes
- Lab 06: add an input and output guardrail layer and measure it
Module 8: Red-Teaming Your Own App
- Building an abuse and test suite for AI features
- Automated prompt injection and jailbreak tests
- Regression testing guardrails and policies
- Running AI security checks in continuous integration
- Model and dependency supply chain: provenance and pinning
- A pre-ship security checklist for AI features
- Lab 07: write automated red-team tests and wire them into CI
Module 9: Scoring AI Security, the SAIS-100 Framework
- The AI Security Hexagon: six questions instead of is it secure
- The six scored categories: data, prompt, agent, supply chain, detection, governance
- The 100-point rubric and its weightings
- Verdict bands and the single-category override rule
- Scoring before and after hardening as a metric
- Scoring exercise against the capstone application
Capstone: Harden an AI Application
- The starter app: an injectable prompt, insecure output handling, unscoped RAG
- The starter app: an over-permissioned agent, secrets in the prompt path, no cost limits
- Restructuring prompts for containment and validating model output
- Sanitizing and scoping retrieval, then applying least privilege and approval gates
- Moving secrets out and adding cost and rate limits
- Adding guardrails and automated red-team tests
- Deliverable: a hardened app plus an OWASP LLM Top 10 self-assessment
Requirements
- Comfort building and consuming REST APIs.
- A scripting language, with labs delivered in Python.
- Basic application authentication, git, and command line familiarity.
- No machine learning background required; this is an application security course for people who build with LLMs, not one that trains them.
Audience
- Software and backend engineers building LLM features.
- Full-stack and API developers.
- AI and ML application engineers.
- Platform engineers shipping copilots and agents.
- Tech leads and senior engineers who own AI features.
Testimonials (3)
AI-ə daxilindəki müxtəlif risk məruz qalmalarını qeydiyyata almaq və müəyyən etmək
Gary Cook - Cybersecurity and Information Technology Risk Division
Course - Introduction to AI Trust, Risk, and Security Management (AI TRiSM)
Machine Translated
I really enjoyed learning about AI attacks and the tools out there to begin practicing and actively using for security testing. I took a lot of knowledge away which I didn't have at the beginning and the course met what I hoped it would be. My favorite part shown from the training was Comet Browser and was amazed at what it could do. Definitely something will be looking into more. Overall it was a great course and enjoyed learning all OWASP GenAI Top 10.
Patrick Collins - Optum
Course - OWASP GenAI Security
The profesional knolage and the way how he presented it before us