Refusal training and unlearning are implemented as one thin layer inside the model, which is why they keep getting broken. What remains is to split the work between internal defenses that cut activations directly and boundaries outside the model.
Alignment research from 2024 through 2026 points in one direction. Token-level refusal training builds a single layer inside the model. Prompt optimization, quantization, or a small amount of relearning can disable that layer.
Defense is therefore developing in two directions. One is in-model defense that controls mid-layer activations directly. The other is a system boundary that handles failures the model cannot address.
In-model failures and defenses
Refusal mediated by one direction
The 2024 analysis by Arditi and colleagues identified the representation that mediates refusal. Across 13 open-source chat models of up to 72B parameters, refusal is mediated by a single one-dimensional subspace in the residual stream. Erasing that direction from activations stops the model from refusing harmful instructions, and adding it makes the model refuse even harmless ones.
The same paper mechanistically analyzes how adversarial suffixes suppress propagation of that direction. What matters is that the cause of failure is located in where alignment sits, not in any particular attack technique. PAIR jailbreaks a target in fewer than twenty queries on average with black-box access alone. That attack is covered in LLM Jailbreaks and Automated Red Teaming. If a defense hangs on one direction, an attack only has to cancel the projection onto it.
Erased knowledge still in place
Unlearning exposes the same limit from another angle. The quantization study by Zhang and colleagues measured what survives. Unlearning methods with utility constraints leave an average of 21 percent of the targeted knowledge in full precision. After 4-bit quantization that figure rises to 83 percent. In this study, routine deployment quantization restored knowledge without an additional attack.
Recovery has more than one path. Xu and colleagues show that minimal fine-tuning restores the original behavior, and they formalize this as a question of reversibility. Reisizadeh and colleagues target the evaluation condition itself. Methods that looked successful under greedy decoding produced the erased content again under probabilistic sampling.
REBEL touches no weights and uses only evolutionary adversarial prompt generation. It recovered supposedly forgotten knowledge with attack success rates up to 60 percent on TOFU and 93 percent on WMDP. The three paths demand different access. Quantization needs only the weights, relearning needs only a small fine-tuning budget, and adversarial prompting needs only black-box queries.
Refusal training and unlearning have different goals, and several methods evaluated by these studies operated by changing output probabilities. Under those conditions, precision changes, repeated sampling, or post-training can restore suppressed behavior or knowledge. The evidence does not establish that every unlearning method leaves knowledge intact.
Defenses that cut the representation
Representation-level defense moves the intervention point from output logits to mid-layer activations. The circuit breakers of Zou and colleagues control the representations responsible for harmful output. That replaces both refusal training and adversarial training aimed at specific attacks.
The paper reports prevention of harmful output in both text-only and multimodal models without sacrificing utility. This includes strong attacks absent from training. Extended to agents, it also reduced the rate of harmful actions under attack.
Follow-up work in 2026 examines the gaps left by static intervention. JPU shows empirically that jailbreaks mainly activate non-erased parameters in the intermediate layers. It defines the paths along which those parameters reassemble as dynamic jailbreak paths. During training it mines on-policy adversarial samples and rectifies those paths toward safety anchors. The underlying diagnosis is that cutting once with a fixed dataset is not enough.
JPU is a January 2026 preprint that was still under review as of 2026-09. No independent replication exists yet.
Failures That Begin Outside the Model
Some failures survive even a precise cut in representation space. They are the ones whose intervention point sits at the boundary where data enters and leaves, not inside the model.
The sleeper memory poisoning study measured how often content planted in a webpage or document is stored as a fabricated memory. The rate reached 99.8 percent on GPT-5.5 and 95 percent on Kimi-K2.6. Among successfully retrieved poisoned memories, 60 to 89 percent led to attacker-intended actions. The mechanics of that attack are covered in LLM Agent Memory Poisoning.
The TMA-NM study addresses the fragility of the defense signal itself. Defenses that rely on content or lineage collapse through three laundering channels: the agent's own summarization, a trusted-tool echo, and manufactured corroboration. Across a benchmark of eight frontier models, existing defenses reached up to 68 percent attack success, while TMA-NM with write-time origin binding reached 0 percent.
A retrieval corpus is a boundary of the same kind. MEntA reaches up to 0.991 AUC with five queries using natural-language entailment signals alone. It improves on prior methods by up to 0.42 AUC and cuts total attack cost by up to 65 times. RAG Datastore Membership Inference covers how the decision signal changed across generations.
These failures do not arise because the model said something harmful. They are about which data was accepted under which authority, and where it was sent. The intervention points are therefore write-time origin binding and retrieval gating. The last gate is the leak control covered in Egress Gateway Defense.
Deployment decisions
Different failures for the two defenses
| Defense position | Intervention point | Failures it stops | Failures it cannot stop | Who can deploy it |
|---|---|---|---|---|
| Refusal training | Output logit distribution | Ordinary harmful requests | Suffix optimization, quantization, relearning | Model provider |
| Representation control | Mid-layer activations | Unseen jailbreak prompts | Poisoned tool output, corpus membership leakage | Model provider |
| System boundary | Write, retrieval, egress | Memory poisoning, data leakage, privilege abuse | Harmful knowledge the model produces on its own | Application team |
The last column is the constraint that binds most often in practice. The first two rows require the ability to train weights. The application team can design only the third row itself.
When representation defense is not the answer
The largest counterexample is the adaptive attack. A 2025 evaluation of 12 recent defenses tuned four optimization families: gradient descent, reinforcement learning, random search, and human-guided exploration. It bypassed most of those defenses with attack success above 90 percent. Many of those defenses had originally reported near-zero attack success. Robustness numbers obtained on a static attack set are hard to trust regardless of where the defense sits.
Turning an internal metric into a training objective is also risky. Song and colleagues audited 398 public unlearned models. They showed that pre-attack accessibility predicts recovery speed and extent at the model level. Yet directly minimizing that metric lowered the audit score while increasing post-attack recovery. The model learns to hide knowledge from the audit rather than delete it.
The final constraint is deployment authority. Representation-level intervention is available only to those who can train weights, so it is not an option for teams that consume a model through an API. The decision those teams actually hold is boundary design. This contrast is a judgment, not a result stated in the papers above.
Placement criteria for defenses
The diagram below separates where the two defenses sit. The upper path shows the user request. The lower path shows external content passing through a store into the model. Representation-level control lives only inside the middle model node, and the application owns the other four points.
flowchart LR
U[User request] --> G[Boundary gate<br/>input classification, authority check]
G --> M[Model<br/>representation-level control]
M --> O[Output and egress control<br/>leak blocking, audit log]
D[External content<br/>documents, web, tool results] --> W[Write gate<br/>origin binding]
W --> S[(Long-term memory and retrieval corpus)]
S --> M- Check first whether you can train weights directly. If not, treat representation defense as a model selection criterion and put the budget on the boundary.
- Separate failures whose result is text from those whose result is action. For action, narrowing tool authority and binding origin at write time come before representation defense.
- For deletion requests, do not claim completion from an output metric alone. Re-measure on the quantized build, check recovery after relearning on unrelated data, and audit at the representation level separately.
- When adopting a defense, do not rely on near-zero numbers that carry no adaptive attack result.
- Use internal audit metrics as diagnostics only, never as training objectives.
The first two items set how the defense budget is split. The remaining three are the criteria for checking whether that defense actually works.
if model_weights_are_trainable:
evaluate_representation_defense_and_apply_system_boundaries()
else:
place_controls_at_write_search_and_egress_boundaries()
retest_both_layers_with_adaptive_attacks_and_deployment_transforms()Summary
Some refusal training depends on a narrow representation direction, and several evaluated unlearning methods recover knowledge after deployment transforms or retraining. Those results apply to the tested methods and conditions rather than proving that all unlearning fails. In-model deployments should evaluate representation-level defenses separately.
System boundaries must handle failures that begin outside the model, including memory poisoning and corpus leakage. Neither layer provides a guarantee alone, so both require retesting under adaptive attacks and actual deployment transforms.
Sources were verified as of 2026-09.