Perturbation Probing: A New Diagnostic for the Fragility of LLM Safety
New research reveals that AI safety refusal lives in a thin neural layer, highlighting the critical need for external, multi-layered security.
New research reveals that AI safety refusal lives in a thin neural layer, highlighting the critical need for external, multi-layered security. Our previous research on logit-gap steering demonstrated that the safety guardrails of an aligned LLM can be bypassed by closing a measurable gap in the model's output scores. The answer matters because it tells defenders whether safety is a thick perimeter or a thin layer of paint. Modern LLMs are aligned through reinforcement learning from human feedback (RLHF), a training stage that pushes the model toward refusing harmful prompts and complying with safe ones.
With only two forward passes per prompt and a significantly lower computational cost, it identifies the small set of feed-forward neurons inside an aligned LLM that are causally responsible for a targeted behavior, such as refusing harmful requests. This concentration matters because it demonstrates that an aligned LLM's refusal behavior does not live in a robust, distributed defense. Across the 13 models tested, this ratio explained 81% of the variance in how vulnerable each model's safety behavior was to a small targeted change. The same toolkit that exposes fragility can also be used to repair it.
On a smaller model, Qwen3.5-2B, just 20 neurons were enough to stop the LLM from falsely agreeing with users in multi-turn conversations, dropping that behavior from 36.7% to 0% across 30 questions. For organizations deploying LLMs today, Prisma AIRS Runtime Security provides the external content filters and inline guardrails that a thin template layer alone cannot.
On open-source LLM Qwen3-4B, just 50 neurons out of 350,208 — about 0.014% of the model's feed-forward neurons — control the safety refusal template. Removing those 50 neurons changes the response format on 80% of 520 standard harmful-prompt benchmarks . Relying on this thin layer alone is the LLM analog of relying on a single perimeter firewall: structurally insufficient. True AI safety demands a defense-in-depth strategy, with external content filters and runtime guardrails layered on top of whatever the base model was trained to do.
What changed
That makes it a candidate for a quantitative safety fragility score , a metric that allows security teams to compare models on alignment robustness without running adversarial red-team campaigns first.
The horizontal axis measures how much a model routes decisions through a narrow internal pathway, and the vertical axis measures how much the model’s safety behavior changed when we disabled just 50 neurons.
Security teams can measure how much of a model's safety rests on a thin, easily removed layer before they put that model in production.
Who is affected
On a smaller model, Qwen3.5-2B, just 20 neurons were enough to stop the LLM from falsely agreeing with users in multi-turn conversations, dropping that behavior from 36.7% to 0% across 30 questions.
For organizations deploying LLMs today, Prisma AIRS Runtime Security provides the external content filters and inline guardrails that a thin template layer alone cannot.
Why this matters
With only two forward passes per prompt and a significantly lower computational cost, it identifies the small set of feed-forward neurons inside an aligned LLM that are causally responsible for a targeted behavior, such as refusing harmful requests.
This concentration matters because it demonstrates that an aligned LLM's refusal behavior does not live in a robust, distributed defense.
Across the 13 models tested, this ratio explained 81% of the variance in how vulnerable each model's safety behavior was to a small targeted change.
The same toolkit that exposes fragility can also be used to repair it.
How organizations are responding
On open-source LLM Qwen3-4B, just 50 neurons out of 350,208 — about 0.014% of the model's feed-forward neurons — control the safety refusal template.
Removing those 50 neurons changes the response format on 80% of 520 standard harmful-prompt benchmarks .
Relying on this thin layer alone is the LLM analog of relying on a single perimeter firewall: structurally insufficient.
True AI safety demands a defense-in-depth strategy, with external content filters and runtime guardrails layered on top of whatever the base model was trained to do.
Beyond identifying the neurons, the same computation produces a diagnostic we call the FFN/Skip ratio: a single number, computable in seconds per model, that predicts whether a model's safety circuit can be easily steered with minimal modifications.
We hope that perturbation probing will serve two roles for the AI security community.
What remains unknown
The available reporting does not establish who is behind the activity, if an attacker is involved.