Harmful Instructions Propensity
Overview
The Harmful Instructions Propensity evaluation measures whether an AI system produces harmful content when it is asked for it plainly, as it would be under realistic use. The system is accessed through its normal input channel, with its own instructions and guardrails in place, and a judge model classifies each response as safe or unsafe.
A high safety score is only meaningful together with the Over-Refusal evaluation, which checks that the system does not achieve it simply by refusing everything.
This is a propensity evaluation: it measures what the system does under ordinary use. Its counterpart, Harmful Instructions Capability, measures how far the same behaviour can be pushed under attack.
Metrics
Safety Score
The fraction of harmful requests whose response the judge classified as safe (range: 0.0 to 1.0, higher is better). It is reported per dataset.
Motivation
Applications built on third-party models, such as question-answering chatbots, inherit the base model's safety training but add their own system prompt, retrieval and guardrails, which can strengthen or weaken it. Plain harmful requests are the most common misuse such systems receive, and need no skill to send.
A system can trivially avoid harmful output by refusing everything, which makes it useless for its intended purpose. Run the Over-Refusal evaluation alongside this one to make that failure visible.
Methodology
- Test cases: harmful requests from StrongREJECT (313 prompts), five HarmBench categories (chemical and biological, cybercrime and intrusion, harassment and bullying, general harm and illegal activities; 186 standard and contextual behaviors) and SOSBench-Lite (300 hazardous science prompts).
- System response: each request is sent to the AI system as a single user message, through its normal input channel. No system message is added. Adapting the input to the system's interface (for example embedding it in a document for a summarisation system) is the job of the model integration, and should be reported with the results.
- Grading: The judge model classifies the response with the Llama Guard safety policy, which covers categories such as violent and non-violent crimes, sex crimes, child exploitation, indiscriminate weapons, hate, self-harm and sexual content.
- Metric computation: safety score per dataset.
Requests that fail because a guardrail returns an error instead of a response leave the judge an empty response, which it classifies as safe.