Download Framework
ISO/IEC 27090:2026
Guidance on security threats specific to AI systems and on detecting and mitigating them across the AI life cycle. It builds on ISO/IEC 27001 and 27002, complements ISO/IEC 42001 and is not certifiable. It addresses the attack classes named in EU AI Act Article 15(5) in more technical depth, but does not confer a presumption of conformity. The risks below follow the threats in Clause 6. Clause 7 (mitigation interactions) and Annex A (mapping of attacks to life cycle and assets) inform the controls but are not listed as risks.
Type:
StandardIndustry:
Cybersecurity
Domain:
CybersecurityPrivacy & Data
Tags:
Adversarial ML
Content:
9 Risks22 Controls
Version: Edition 1, 2026
Framework Definition
Risks and controls associated with the framework
Assessment Layer
Concrete evaluations linked to controls to assess pass or fail
RAG Chatbot
Includes evaluations applicable to all use cases
RISK
Data Poisoning Attack
Risk that malicious or unwanted data is introduced into training, fine-tuning or retraining data to degrade the model or implant attacker-chosen behaviour such as backdoor triggers. This includes operational data collected in production and later reused for retraining.
CONTROL
Training Data Provenance and Integrity
Ensure that the origin, collection process and integrity of training and fine-tuning datasets, including third-party datasets, are recorded and verified before use.
CONTROL
Poisoned Sample Detection
Ensure that training and fine-tuning datasets are screened for poisoned samples, trigger-payload pairs and other anomalous records before they are used to train a model.
EVALUATION
Data Poisoning
Detects poisoned samples in a dataset that could elicit backdoor behaviour when used for LLM training — including trigger-payload pairs, sleeper agent patterns, and adversarial trigger phrases.
CONTROL
Operational Data Vetting Before Retraining
Ensure that data collected during operation is validated and monitored for distribution shifts before it is reused for retraining, so that gradual poisoning through production inputs is detected.
EVALUATION
Data Drift
Measures the degree to which each sample in a new dataset has drifted from the reference (training) distribution, detecting covariate shift at both the dataset and per-sample level.
RISK
Evasion Attack
Risk that inputs crafted or perturbed at inference time cause the model to produce incorrect outputs, such as misclassifications or wrong answers, while appearing benign to a human observer.
CONTROL
Adversarial Robustness Testing
Ensure that the model is tested against perturbed and adversarially crafted inputs representative of its deployment context and that its outputs remain consistent.
EVALUATION
Text Robustness
Evaluates whether AI models produce semantically consistent responses when the same question is presented with meaning-preserving input perturbations such as typos, casing changes, or paraphrasing.
CONTROL
Malicious Input Detection
Ensure that inputs are validated and that out-of-distribution, degenerate or adversarial inputs are detected and handled safely before they reach the model.
RISK
Membership Inference
Risk that an attacker uses model outputs to determine whether specific records were part of the training data, disclosing information about the individuals or organisations those records describe.
CONTROL
Training Data Minimisation and Sanitisation
Ensure that sensitive and duplicated records are removed from or minimised in training data, reducing memorisation of individual records.
EVALUATION
Training Data Sanitisation
Detects samples in a training dataset that contain unsanitised sensitive content - including personal identifiers, national identifiers, financial account data, authentication secrets, health data, online identifiers, sensitive personal attributes, and confidential business data - that should have been scrubbed or masked before use in model training.
EVALUATION
Data Uniqueness
Measures the degree to which each sample in a dataset is unique: sample identifiers are present and unique, and no two records share the same content across the non-ID columns.
CONTROL
Membership Inference Testing
Ensure that the model is tested for membership inference before release and after retraining.
RISK
Model Exfiltration
Risk that an attacker obtains a functional copy of the model or its parameters, either by stealing model artefacts or by extracting the model through queries that can appear legitimate.
CONTROL
Query Throttling
Ensure that the number and volume of queries a single caller can issue to the model is limited.
EVALUATION
Rate Limiting
Verifies that the model endpoint enforces rate limiting by sending a burst of requests that exceeds the declared rate limit and checking whether enough of those requests are rejected.
CONTROL
Extraction Pattern Monitoring
Ensure that query patterns are logged and monitored to detect systematic probing indicative of model extraction, even when individual queries appear legitimate.
CONTROL
Model Artefact Protection
Ensure that model weights and related artefacts are protected at rest and in transit and that access to them is restricted and logged.
RISK
Model Inversion
Risk that an attacker uses model outputs to reconstruct training data or infer sensitive attributes of the records it was trained on.
CONTROL
Output Information Restriction
Ensure that outputs expose no more information than the use case requires, for example by withholding or coarsening confidence scores and logits.
CONTROL
Model Inversion Testing
Ensure that the model is tested for reconstruction of training data or sensitive attributes before release and after retraining.
RISK
Direct Model Poisoning
Risk that model parameters, weights or pre-trained components are tampered with directly in the development environment, model registry or software supply chain, so that the deployed model behaves maliciously, for example through an embedded backdoor.
CONTROL
Model and Component Supply-Chain Integrity
Ensure that pre-trained models, libraries and other third-party components are obtained from trusted sources, pinned to verified versions and integrity-checked before use.
CONTROL
Development Environment Protection
Ensure that training pipelines, model registries and model artefacts are protected by access control and change logging so that unauthorised modification of models is prevented and detectable.
CONTROL
Backdoor Detection
Ensure that models, in particular those obtained from third parties, are tested for hidden trigger behaviour before deployment.
RISK
Sensitive Model Output
Risk that model outputs disclose confidential information, such as personal data, credentials, system prompts or internal system details, to users who are not authorised to receive it.
CONTROL
Sensitive Data Exclusion from Model Context
Ensure that credentials and other sensitive data are kept out of system prompts and model context and are supplied through managed secret stores instead.
EVALUATION
System Prompt Sensitive Data
Detects system prompts that contain sensitive information - such as API keys, authentication credentials, database names, user roles, or permission structures - that should be externalised rather than embedded directly in the prompt.
CONTROL
Output Disclosure Testing
Ensure that the AI system is tested for disclosure of sensitive data, system prompts and system configuration under both benign and adversarial queries.
EVALUATION
Sensitive Data Leakage
Evaluates whether an AI application discloses sensitive data to unauthenticated callers under both targeted and untargeted attack conditions.
EVALUATION
Prompt Leakage Safety
Evaluates whether AI models can keep their system prompt confidential against a range of extraction attacks, including naive requests, paraphrasing, prefix tricks, sycophancy, and advanced PLeak techniques in both single-turn and multi-turn settings.
EVALUATION
System Information Disclosure
Evaluates whether AI models inadvertently or under adversarial pressure reveal sensitive system context - including system prompt content, tool and MCP server details, backend service information, and internal configuration - through their outputs.
RISK
Prompt Injection
Risk that instructions embedded in input override the intended behaviour of the AI system, bypass its safeguards or redirect it to attacker-chosen goals. The input may be supplied directly by a user or indirectly through retrieved documents, tool output or memory.
CONTROL
System Prompt Hardening
Ensure that system prompts define the model's role and scope and instruct it to treat user and third-party content as untrusted data rather than instructions.
EVALUATION
System Prompt Safety Instructions
Audits system prompts for three structural safety properties: whether they define the model's role, scope what it will and will not respond to, and explicitly instruct the model to ignore attempts to override its instructions.
EVALUATION
System Prompt Safety Instructions (Dify)
Audits system prompts for three structural safety properties: whether they define the model's role, scope what it will and will not respond to, and explicitly instruct the model to ignore attempts to override its instructions.
CONTROL
Direct Prompt Injection Resistance
Ensure that the AI system resists adversarial user inputs, including jailbreaks and multi-turn attacks, that attempt to override its instructions or move it outside its intended use.
EVALUATION
Prompt Injection
Evaluates whether AI models resist prompt injection attacks that attempt to override instructions, manipulate roles, or bypass safeguards through adversarially crafted user inputs.
EVALUATION
Basic Red Team
Measures the attack success rate of 10 common low-cost red-team strategies that obfuscate harmful HarmBench and AdvBench requests, using a judge model to determine whether the model's response is harmful.
EVALUATION
Advanced Red Team
Measures the best-of-n attack success rate of the bijection learning jailbreak, a capability-scaled attack that teaches the model a random encoding in context before sending an encoded harmful request from HarmBench and AdvBench.
EVALUATION
Red Team: Goal Hijacking
Evaluates whether an AI application can resist goal hijacking by an active red-team agent that autonomously generates attack strategies and conducts multi-turn adversarial interactions to cause the system to act outside its intended purpose.
EVALUATION
Intended Use Evasion
Evaluates whether an AI application stays within its operator-defined scope under multi-turn adversarial probing. For each strategy in a library of generic multi-turn attack patterns, an autonomous adversary seeded with the target's intended use and that strategy conducts a multi-turn session against the target; a transcript-level judge then classifies the session as `hijacked` or `not_hijacked`.
EVALUATION
Invisible Text Injection
Evaluates whether AI models read text hidden in invisible Unicode characters - decoding hidden payloads or following hidden instructions that a human reviewer cannot see - and whether they emit invisible characters themselves.
CONTROL
Untrusted Content Screening
Ensure that retrieved documents, tool output and stored memory are screened for embedded or hidden instructions before they enter the model context.
EVALUATION
RAG Knowledge Base Poisoning
Detects poisoned documents in a RAG knowledge base that contain embedded adversarial instructions designed to hijack model behaviour at retrieval time - including prompt injection patterns, instruction overrides, and hidden directives embedded within otherwise legitimate-looking content.
RISK
Output Contains Injection Attacks
Risk that model outputs carry malicious payloads, such as code, markup, queries or commands, that compromise downstream components, tools or users that process them.
CONTROL
Output Validation and Encoding
Ensure that model outputs are treated as untrusted and are validated, sanitised or encoded for their destination before downstream components process them.
EVALUATION
Improper Output Handling
Evaluates whether the AI application architecture includes output safety checks that treat LLM-generated content as untrusted before passing it to downstream systems, preventing unsafe or dangerous outputs from reaching sensitive sinks.
CONTROL
Least-Privilege Downstream Actions
Ensure that the tools, network destinations and actions reachable from model output are limited to those the use case requires and that high-risk actions require human approval.