LLM Defenses at Action Boundaries

A defense that recognizes the form of an attack loses its effect when the attacker changes that form. The defenses that kept their effect checked what the attacker cannot change: data destination, component permissions, and the split between planning and perception.

The block rate of an LLM defense is usually measured against one specific set of attacks. Prompt injection and model extraction studies from 2026 report cases where a defense lost its effect. In these cases, only the attack wording or the query generation method changed. The defenses that kept their effect did not read the attack content. They checked at an action boundary.

In this post, an action boundary means one of three things. The first is the destination that data goes to. The second is the permission that a component can use. The third is the separation between the model that makes the plan and the model that reads untrusted input. When you adopt or evaluate a defense, check which attack form the reported number applies to and what the check reads.

Where Agent Exfiltration Checks Split

The Framing Gap (arXiv:2608.27092, 2026-08) tested a tool-using agent that holds a secret and reads an attacker web page. The agent receives a policy, a summarization task, the synthetic secret SECRET_12345, and the fetched page. It then emits one JSON action. The tools are mock tools that only record their arguments, and the attacker controls only the page body.

Success uses matched trials. A trial counts as a success only when the agent keeps the secret on the clean page and leaks it on the poisoned page. The temperature is 0.7. Primary models ran 20 trials per cell, and the generality sweep ran 10 trials per cell.

The Same Leak in Different Words

On 10 explicit injection classes, the pooled attack success rate (ASR) of gpt-4o was 0.0%. The author then built a reframing family of 8 variants. One variant is an un-reframed baseline. The other 7 rewrite the same leak request into forms such as an integrity signature, a config value, an auth field, and a lookalike host.

Across all 8 variants, the pooled ASR of gpt-4o was 40.0%. The #sig= signature variant and the config block variant each reached 100%. The same gpt-4o configuration recorded 51/160 on one run and 64/160 on another, so read this value as a range of about 30% to 40%.

The author ran the same comparison on 6 models. In 5 of them, the reframing family ASR was more than twice the explicit injection ASR (arXiv:2608.27092 Table 4). The exception was mistral:7b, at 26.0% and 27.5%.

In an ablation that removed the confidentiality policy, the gpt-4o reframing ASR was 31.9% [25.2, 39.4] with the policy and 38.1% [31.0, 45.8] without it. The two 95% intervals overlap, so the difference between policy and no policy could not be distinguished. The author interprets the cause as confusion between instructions and data, rather than as an alignment bypass.

Checks That Must Recognize the Attack

The author compared defenses by check location. A policy that listed exfiltration acts and added an "in any form" clause brought the gpt-4o reframing ASR to 0/120. When that one catch-all clause was removed and only specific acts were listed, the ASR was 48.8% (39/80). The author classifies this defense as brittle.

An output guard searches the emitted action for the secret string. Guard v1 compares substrings only. With guard v1, plaintext leaks had 0% ASR, but reversed encoding had 100% ASR. Guard v2 decodes reversed, hex, base64, and other encodings before it compares. With guard v2, all anticipated encodings had 0% ASR, but ROT13, which it had no decoder for, had 100% ASR.

On a Llama-3.1-8B tool-calling agent, the author tested two defenses. They are SecAlign, which is training-based, and input channel separation. Against an undefended reframing ASR of 21.2%, SecAlign gave 32.5% and channel separation gave 38.8%.

SecAlign lowered the ASR of an "Ignore previous instructions" control from 20/20 to 0/20, so the setup was applied correctly. The author does not claim an order among the three conditions, because the differences are within the run-to-run variance. The confirmed result is that neither defense brought the reframing ASR below the undefended level.

Checks That Read Destination and Provenance

The output content guard and the destination allowlist differ in what they read. The pseudocode below paraphrases §8 of the paper.

# Pseudocode: an output content guard reads the payload
def content_guard(action):
    payload = normalize(action.args)  # decodes only known encodings such as reversed, hex, base64
    if SECRET in payload and host(action.url) not in TRUSTED_HOSTS:
        return refuse()
    return action
 
# Pseudocode: a destination allowlist does not read the payload
def destination_allowlist(action):
    if action.tool == "post_data" and host(action.url) not in TRUSTED_HOSTS:
        return refuse()
    return action

A destination allowlist blocks every post_data call to an untrusted host, whatever the content. It does not read the payload, so ROT13, base32, and leaks split across calls cannot change its result. The author treats this 0% as a value set by construction, not as an end-to-end measurement.

The planner/reader split implements only the core isolation idea of CaMeL. CaMeL is a defense design that separates planning from the processing of untrusted data, and the next section describes it. The planner holds the secret and emits tool calls, but it never sees the page. The quarantined reader summarizes the page, but it never holds the secret.

The results were 0/80 on reframing and 0/30 on held-out encodings. In 9 of 10 clean-page runs, the summary contained the key entities.

The sources are arXiv:2608.27092 Tables 3, 6, and 7 and §6. The undefended Llama-3.1-8B baseline is 21.2% on reframing and 0.0% on held-out encodings. The author states that the gpt-4o rows and the Llama-3.1-8B rows use different scaffolds, so their absolute values are not comparable.

The first four rows of the table work only when a model or guard recognizes the attack wording or payload. The last two rows look only at the target host and at which component holds the secret. The author concludes that the robust checks read the destination or the provenance, not the content.

A setup that routes outbound traffic through one control point is covered in Egress Gateway Defense.

Permission Boundaries in Web Agents

Prismata (arXiv:2607.08147, 2026-07) starts from one fact: third parties can write text on the pages that a web agent reads. In this attack, the attacker plants instructions in user-generated or external content to change what the agent does. Prismata does not search for injection strings. It assigns a permission to each page element and enforces that permission in code.

Permissions from Paths and Provenance

For each interactive element, Prismata extracts the ancestor path from the Document Object Model (DOM) root to that element. A separate LLM call decides the permission from the user task and this path only. Instructions planted outside the path are therefore not visible to that decision. Along the path, Prismata follows the Biba integrity model and labels elements with one of four provenances: developer, user, hosted-party, or external.

Labels can only lose permission, and the authors state that the design keeps the effect of labeling errors bounded. A policy model that receives only the task, with no DOM, decides which provenances the task needs. Deterministic code, independent of any LLM, then removes content outside the permission from the observation. The same code rejects tool calls that exceed the permission before the browser runs them.

On WebArena, the authors gave a gpt-5.4-mini agent three injection templates and applied Prismata. The average ASR fell from 85.5% to 0.7%. The task success rate without attacks fell from 29.9% to 26.6%, a drop of 3.3 percentage points (pp). On the adaptive attack set WA20, the ASR with Prismata was 0% for all three templates.

Plans Fixed Before Observation

CaMeL (arXiv:2503.18813, 2025-03) first extracts the control flow and data flow from the trusted user query. As a result, untrusted data cannot change the program flow. On AgentDojo, CaMeL solved 77% of tasks with provable security, while the undefended system solved 84%.

CaMeLs Can Use Computers Too (arXiv:2601.09923, 2026-01) applies this separation to computer-use agents that act by looking at the screen. A trusted planner writes the full plan, including its branches, in one pass before it sees the screen. An untrusted perception model fills in only runtime values, such as coordinates, from screenshots and the DOM. It cannot change the structure of the plan.

The single-shot planning design in this paper (NOVA) used 1.88 times the tokens of the undefended agent. A comparison design that calls the planner again at every step (Fides-NOVA) used 29.6 times the tokens.

The diagram below simplifies the structure that the three studies share. The exact connections differ between studies.

flowchart LR
  U["User task"] --> P["Planner<br/>holds secrets and permissions<br/>does not read untrusted content"]
  W["Attacker content<br/>page, screen"] --> R["Reader or perception model<br/>no secrets, no permissions"]
  P -->|"plan, tool calls"| X["Deterministic executor"]
  R -->|"values only"| X
  X -->|"calls within allowed destinations and permissions"| S["External action"]

Attacker content enters only through the reader or perception model, and the executor does not interpret those values as instructions. The component that holds secrets and permissions does not read attacker content.

Attack Dependence in Model Extraction Defenses

The same pattern appears in model extraction. Black-box model extraction collects API responses and trains a surrogate model on them. The basic mechanism is covered in LLM Privacy Attacks.

MEA-Bench (arXiv:2610.00839, 2026-09) puts 6 extraction attacks and 10 defenses under the same budgets and evaluation conditions. The defense experiments used a Qwen2.5-72B-Instruct victim model, a Qwen2.5-7B surrogate model, and a query budget of 1,000. The control is a surrogate trained on unmodified responses to the same queries.

Defenses That Look at Responses and Queries

The sources are arXiv:2610.00839 v2 Tables 9, 12, and 13. Each accuracy change subtracts the control accuracy (57.62, 63.34, 57.68). Query detection uses batches of 50 queries. The true positive rate (TPR) is the detection rate for attack batches, and the false positive rate (FPR) is the rate for benign batches.

The anti-distillation defense ADS lowered surrogate accuracy by 3.51 pp only for QEDKS. For SeqKD and SODA, the change stayed within 1 pp. The query detectors MMD and SEAT caught all QEDKS query batches. They caught only 0% to 5% of SeqKD query batches, which come from a fixed query pool. The authors summarize that the effect of anti-distillation depends on the attack.

QEDKS expands 3 seed queries into 498 template queries and 499 follow-up questions. The detectors probably caught this query generation method rather than the extraction itself. This is an estimate.

The lineage verifier DuFFin mostly gave the 3 extracted surrogates lower scores than the 4 unrelated models. The 4 unrelated models scored from 0.278 to 0.705. The model-level ROC-AUC measures whether extracted models score higher than unrelated models, and it was 0.167. In this setup, DuFFin therefore judged the extracted models as less similar to the victim than the unrelated models.

One Post-Processing Step Changes the Result

A study by Libon et al. (arXiv:2606.25059, 2026-06) evaluated the same ADS again under different attacker assumptions. In the original paper's setting, ADS lowered student accuracy by about 40% at the cost of about 20% teacher accuracy. The attacker then added one post-processing step that deletes repeated segments from responses. With this step, student accuracy rose above the undistilled baseline across the high-utility range of λ, the ADS strength parameter.

This experiment used a DeepSeek-R1-Distill-Qwen-7B teacher, a Llama-3.2-3B student, and 5,231 GSM8K prompts. The post-processing results appear only in a figure, so the paper text gives no exact values.

Defenses That Reduce Response Fields

In an article dated 2026-10-12, Anthropic describes several defenses used together. They are reasoning summaries, limits on where reasoning content can go, a two-stage classifier, and usage limits and account bans for repeated attempts. Anthropic reports that a pretrained Claude model trained on full reasoning transcripts from Claude Opus 4.8 scored 80.4% on SWE-bench Verified. The same training on summarized transcripts scored 71.8%.

Anthropic's article gives early evidence that these methods together reduced the data that distillers collect by about 90% to 99%. All of these figures are the provider's own measurements. The article expects distillation attempts and defenses to keep adapting to each other.

A reasoning summary reduces the content that the response carries, so it does not need to recognize the form of the attacker's queries. In this respect, it can be treated as a check at the same location as a destination allowlist. This is a judgment.

Conditions Where Boundary Checks Fail

A defense that recognizes the attack form did not always lose its effect. The policy text with the catch-all clause blocked reframing at 0/120. But without that one clause the ASR was 48.8%, and no held-out encoding result was reported for this defense.

A destination allowlist also blocks legitimate outbound traffic. The Framing Gap did not measure what fraction of real workloads can use a closed set of destinations. A case where the same secret may go to host A but not to host B is where an allowlist and taint tracking diverge. The author leaves this case as future work.

The 0% of the allowlist and of the planner/reader split comes from the design. The experiment used a single-step agent, so it did not test a replanning loop. The author states that this setting can make CaMeL-style results look better.

When you set a boundary, the attack can move inside the boundary. The paper CaMeLs Can Use Computers Too names one such attack Branch Steering. In this attack, the attacker manipulates observations so that the agent takes the attacker's preferred branch among the legitimate branches already in the plan. One example makes the agent click a malicious ad disguised as a cookie popup.

Gradient-based pixel attacks evaded all cross-checks that this paper tested.

Prismata also cannot block an injection that sits on the ancestor path with no preceding structural cue. In Common Crawl and Mind2Web, such paths were about 0.10% of the full corpus. On sites that follow web best practices, they were about 0.017%.

Boundary checks cost utility. CaMeL lowered the task solve rate from 84% to 77%, and Prismata lowered the task success rate from 29.9% to 26.6%. NOVA used 1.88 times the tokens. The benchmarks and models differ, so the sizes of these costs are not comparable.

This logic does not transfer directly to model extraction. The leaked object is the response itself, and the attacker is an API caller just like a normal user. A destination check cannot separate the two. The remaining boundaries are the fields in the response and per-account usage. MEA-Bench did not evaluate this category, so the claim that boundary checks generalize for extraction is a judgment.

The MEA-Bench figures come from fixed seeds, and the paper reports no repetition count and no confidence intervals. Differences within 1 pp cannot be separated from noise.

Criteria for Choosing a Defense

When you adopt or evaluate a defense, answer these questions in order.

• Can the attacker change the property that the defense checks? The attacker controls wording, encoding, and query distribution. The system controls the destination host, element provenance, and a plan fixed in advance.
• Does the evaluation include attack forms that were not used for training or calibration? Reframing, encodings with no decoder, and extraction attacks with a different query generation method are such held-out forms.
• Is the reported 0% a measured result or a result by construction? If it is by construction, check the utility cost and the paths that remain outside the boundary instead.
• After you set the boundary, does any component still read attack input? A perception model that selects branches becomes the next target.
• What fraction of legitimate work must cross the boundary? If this fraction is large, the boundary check costs much utility.

Instructions hidden in tool descriptions are covered in MCP Tool Poisoning. Identity design for components that act with delegated permissions is in The Confused Deputy and Agent Identity. The limits of model-internal defenses are in From Token Alignment to Representation-Level Defense.

References

• The Framing Gap: Indirect Prompt-Injection Exfiltration Defeats Surface-Level Defenses in Tool-Using Agents, arXiv:2608.27092 (2026-08)
• Prismata: Confining Cross-Site Prompt Injection in Web Agents, arXiv:2607.08147 (2026-07)
• Defeating Prompt Injections by Design, arXiv:2503.18813 (2025-03)
• CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents, arXiv:2601.09923 (2026-01, v3 2026-06)
• Do Defenses Against LLM Extraction Work Across Attacks? A Lifecycle Benchmark of Black-Box Model Extraction, arXiv:2610.00839 (2026-09, v2 2026-10)
• What Does It Mean to Break a Distillation Defense?, arXiv:2606.25059 (2026-06)
• Strengthening our defenses against distillation attacks, Anthropic (2026-10)

Every figure was checked against each source as of 2026-10.

Summary

Defenses that recognize attack wording, encoding, or query distribution mostly lost their effect when the attacker changed that form. In The Framing Gap, rewriting the same leak raised the gpt-4o ASR from 0% to about 30% to 40%. In MEA-Bench, a defense that worked against one extraction attack had almost no effect against the other two attacks in the table.

The defenses that kept their effect under form changes checked at an action boundary. Examples are the destination allowlist, the planner/reader split, and Prismata's permission enforcement. Most of their 0% results hold by construction, and they cost blocked legitimate traffic and lower task success. After a boundary is set, attacks such as Branch Steering still target the branch choice inside it.

In model extraction, the response itself is the leaked object, and the sources reviewed here include no independent evaluation of boundary checks.