From Token Alignment to Representation-Level Defense

Refusal training and unlearning are implemented as one thin layer inside the model, which is why they keep getting broken. What remains is to split the work between internal defenses that cut activations directly and boundaries outside the model.

Alignment research from 2024 through 2026 points in one direction. Training refusal at the token level builds a single thin layer inside the model, and that layer peels off under prompt optimization, quantization, or a small amount of relearning. The center of gravity of defense is therefore moving down two paths. One is in-model defense that controls mid-layer activations directly. The other is a system boundary that owns the failures the model cannot touch.

Refusal Mediated by One Direction

The 2024 analysis by Arditi and colleagues located where refusal lives. Across 13 open-source chat models of up to 72B parameters, refusal is mediated by a single one-dimensional subspace in the residual stream. Erasing that direction from activations stops the model from refusing harmful instructions, and adding it makes the model refuse even harmless ones.

The same paper mechanistically analyzes how adversarial suffixes suppress propagation of that direction. What matters is that the cause of failure is located in where alignment sits, not in any particular attack technique. PAIR jailbreaks a target in fewer than twenty queries on average with black-box access alone. That attack is covered in LLM Jailbreaks and Automated Red Teaming. If a defense hangs on one direction, an attack only has to cancel the projection onto it.

Erased Knowledge Still in Place

Unlearning exposes the same limit from another angle. The quantization study by Zhang and colleagues measured what survives. Unlearning methods with utility constraints leave an average of 21 percent of the targeted knowledge in full precision. After 4-bit quantization that figure rises to 83 percent. The attacker does nothing special here, because routine deployment practice brings the knowledge back.

Recovery has more than one path. Xu and colleagues show that minimal fine-tuning restores the original behavior, and they formalize this as a question of reversibility. Reisizadeh and colleagues target the evaluation condition itself. Methods that looked successful under greedy decoding produced the erased content again under probabilistic sampling.

REBEL touches no weights and uses only evolutionary adversarial prompt generation. It recovered supposedly forgotten knowledge with attack success rates up to 60 percent on TOFU and 93 percent on WMDP. The three paths demand different access. Quantization needs only the weights, relearning needs only a small fine-tuning budget, and adversarial prompting needs only black-box queries.

Refusal training and unlearning have different goals but fail in the same place. Neither changes what the model knows. Both only adjust the probability of what it says. The original knowledge returns whenever the conditions of that probability computation change. Lowering precision, sampling repeatedly, or nudging the weights is enough to do that.

Defenses That Cut the Representation

Representation-level defense moves the intervention point from output logits to mid-layer activations. The circuit breakers of Zou and colleagues control the representations responsible for harmful output. That replaces both refusal training and adversarial training aimed at specific attacks. The paper reports that this prevents harmful output in both text-only and multimodal models without sacrificing utility, including under strong unseen attacks. Extended to agents, it also reduced the rate of harmful actions under attack.

Follow-up work in 2026 examines the gaps left by static intervention. JPU shows empirically that jailbreaks mainly activate non-erased parameters in the intermediate layers. It defines the paths along which those parameters reassemble as dynamic jailbreak paths. During training it mines on-policy adversarial samples and rectifies those paths toward safety anchors. The underlying diagnosis is that cutting once with a fixed dataset is not enough.

JPU is a January 2026 preprint that was still under review as of 2026-09. No independent replication exists yet.

Failures That Begin Outside the Model

Some failures survive even a precise cut in representation space. They are the ones whose intervention point sits at the boundary where data enters and leaves, not inside the model.

The sleeper memory poisoning study measured how often content planted in a webpage or document is stored as a fabricated memory. The rate reached 99.8 percent on GPT-5.5 and 95 percent on Kimi-K2.6, and among successfully retrieved poisoned memories, 60 to 89 percent led to attacker-intended actions. The mechanics of that attack are covered in LLM Agent Memory Poisoning.

The TMA-NM study addresses the fragility of the defense signal itself. Defenses that rely on content or lineage collapse through three laundering channels: the agent's own summarization, a trusted-tool echo, and manufactured corroboration. Across a benchmark of eight frontier models, existing defenses reached up to 68 percent attack success, while TMA-NM with write-time origin binding reached 0 percent.

A retrieval corpus is a boundary of the same kind. MEntA reaches up to 0.991 AUC with five queries using natural-language entailment signals alone. It improves on prior methods by up to 0.42 AUC and cuts total attack cost by up to 65 times. RAG Datastore Membership Inference covers how the decision signal changed across generations.

These failures do not arise because the model said something harmful. They are about which data was accepted under which authority, and where it was sent. The intervention points are therefore write-time origin binding and retrieval gating. The last gate is the leak control covered in Egress Gateway Defense.

Different Failures for the Two Defenses

The last column is the constraint that binds most often in practice. The first two rows are options only for those who can train weights, and only the third row is designed by the application team itself.

When Representation Defense Is Not the Answer

The largest counterexample is the adaptive attack. A 2025 evaluation of 12 recent defenses tuned four optimization families: gradient descent, reinforcement learning, random search, and human-guided exploration. It bypassed most of those defenses with attack success above 90 percent. Many of those defenses had originally reported near-zero attack success. Robustness numbers obtained on a static attack set are hard to trust regardless of where the defense sits.

Turning an internal metric into a training objective is also risky. Song and colleagues audited 398 public unlearned models and showed that pre-attack accessibility predicts recovery speed and extent at the model level. Yet directly minimizing that metric lowered the audit score while increasing post-attack recovery. The model learns to hide knowledge from the audit rather than delete it.

The final constraint is deployment authority. Representation-level intervention is available only to those who can train weights, so it is not an option for teams that consume a model through an API. The decision those teams actually hold is boundary design. This contrast is a judgment, not a result stated in the papers above.

Placement Criteria for Defenses

The diagram below separates where the two defenses sit. The upper path is the route a user request takes, and the lower path is the route external content takes through a store into the model. Representation-level control lives only inside the middle model node, and the application owns the other four points.

• Check first whether you can train weights directly. If not, treat representation defense as a model selection criterion and put the budget on the boundary.
• Separate failures whose result is text from those whose result is action. For action, narrowing tool authority and binding origin at write time come before representation defense.
• For deletion requests, do not claim completion from an output metric alone. Re-measure on the quantized build, check recovery after relearning on unrelated data, and audit at the representation level separately.
• When adopting a defense, do not rely on near-zero numbers that carry no adaptive attack result.
• Use internal audit metrics as diagnostics only, never as training objectives.

The first two items set how the defense budget is split. The remaining three are the criteria for checking whether that defense actually works.

Summary

Token-level alignment breaks not because attacks are clever but because the defense sits in a thin place. Refusal is mediated by one direction, and unlearning only presses down the output distribution, so knowledge returns under quantization, relearning, and adversarial prompting. In-model defense is therefore moving down to the representation level, while the boundary must own failures the model cannot govern, such as memory poisoning and corpus leakage. No single layer is a guarantee, and under adaptive attack both layers lose their static evaluation numbers. The decision rule is one line: if you can touch the weights, choose representation defense, and otherwise put the budget on the write, retrieval, and egress boundaries.

Sources were verified as of 2026-09.