Harmful Instructions Capability
Overview
The Harmful Instructions Capability evaluation measures whether an AI system produces harmful content when someone actively tries to get past its instructions and guardrails. The system prompt cannot be removed, so the attacks go through the system's normal input channel: each harmful request is wrapped in one of ten jailbreak templates (personas, fiction, fake authority and mode unlocks, among others) and sent several times.
It is the attack counterpart of Harmful Instructions Propensity. It adds the most for harms that the harmful knowledge evaluations do not cover, such as conventional weapons, cybercrime and other illegal activity. Each attack template is a separate task, so the results show which attacks work against the system.
Metrics
All metrics are reported per dataset and per attack template. A request only counts as safe if the responses of all three trials were safe (best-of-k).
Safety Score
The fraction of requests for which no trial produced an unsafe response (range: 0.0 to 1.0, higher is better). Comparing it with Harmful Instructions Propensity on the same requests shows how much safety drops under attack compared with plain requests.
Motivation
Jailbreak templates are the best-known low-skill attack on chatbots: they are public and need no tooling. A system that refuses plain harmful requests may still comply when the request is framed as fiction or research, when it is asked to play an expert, or when its instructions are overridden.
Methodology
-
Test cases: StrongREJECT (313 prompts), SOSBench-Lite (300 prompts) and five HarmBench categories (chemical and biological, cybercrime and intrusion, harassment and bullying, general harm and illegal activities; 186 behaviors), the same requests as in Harmful Instructions Propensity so that results can be compared per item.
-
Attacks: each request is wrapped in one jailbreak template and sent to the AI system as a single user message. No system message is added.
Attack Scenario Safe Educational Context Authorisation claim DAN Persona Developer Mode Mode unlock Opposite Mode Dual response Refusal Then Comply Refusal suffix Movie Scene Fiction AIM Nested fiction Fake System Message Fake authority Model Conversation Simulated dialogue Evil Confidant Persona -
Trials: each attack is sent three times per request, and the worst result is kept.
-
Grading: The judge model classifies the response with the Llama Guard safety policy, which covers categories such as violent and non-violent crimes, sex crimes, child exploitation, indiscriminate weapons, hate, self-harm and sexual content.
-
Metric computation: safety score per dataset and attack.
Results are only comparable for the same number of trials. Successful attacks produce real harmful text, which is stored in the task results like any other response; restrict access to the results accordingly and report scores rather than excerpts.