How Personal Data Leaks, and What Data Governance Has to Cover

Securing external LLMs and agents starts by separating the leak paths, then using data tiers to limit what may be sent where.

Why the Threat Model Comes First

Calling an external LLM API hands identifying data such as membership records, payments, and medical information to a vendor's infrastructure. That makes the call a processing delegation. Treating it as just another function call means missing the leak paths.

The data flow has to be settled before the model choice. The starting point is knowing which data travels which path to whom.

A token masking library on its own is not enough. Even if names and phone numbers are stripped from the input, the original values remain in the RAG context and tool call arguments attached to the same request.

Observability traces are a separate path. Logs kept for debugging become the personal data store that survives the longest.

This post separates Personally Identifiable Information (PII) leakage into four paths. It then identifies which infrastructure may receive data at each tier.

The Paths PII Leaks Through

Each path differs in how long data stays and in what medium. Input lives for the duration of one request, while observability logs are retained for months or years. The same PII has a different exposure surface on each path, so the defenses differ too.

One Query, Four Paths

Assume the following query arrives at a hypothetical internal chatbot.

Summarize the 5 most recent transactions for member Hong Gildong (010-1234-5678).

That single line leaves a trace on all four paths.

1. Input path: the prompt goes into the external LLM API body as-is. Hong Gildong and the phone number arrive verbatim on OpenAI or Anthropic servers.
2. RAG/tool path: when the transaction DB lookup tool runs, the result rows can drag other members' identifiers into the context as well.
3. Output path: as the model reconstructs an answer in the form "Hong Gildong's May transactions were...", the masked PII reappears in the output.
4. Observability path: LangSmith or an in-house tracer may store the input, tool results, and output verbatim. That store then holds all the PII in one place.

One piece of PII sits in four places at once. Blocking a single point does not stop the leak.

OWASP Top 10 for LLM Applications 2025 is a list of the leading security risks in LLM applications. It is compiled by the nonprofit security organization OWASP (Open Worldwide Application Security Project). Think of it as the LLM version of the web Top 10. The 2025 edition (v2.0, published 2024.11) consists of ten entries, LLM01 through LLM10.

Three entries in the OWASP Top 10 for LLM map directly onto those four paths.

• LLM01 Prompt Injection: the trigger that enters through (1) and (2) and exits through (3) and (4)
• LLM02 Sensitive Information Disclosure: the output re-leak of (3) itself
• LLM06 Excessive Agency: amplifies the risk in (2) when tool permissions are too broad

Model and infrastructure paths

Beyond the four application paths, evaluate exposure conditions for models and stores. The following studies test different access conditions.

• Embedding inversion: Vec2Text, published at EMNLP 2023, evaluates text reconstruction from target vectors. Its results apply to the tested embedding models and inputs. They support treating vectors as potentially revealing, not automatically anonymous.
• Training-data extraction: Carlini's USENIX Security 2021 study verifies extraction of training strings from GPT-2. It does not establish that a particular user's API conversations were trained on or describe a current provider's retention policy.
• External tool responses: InjecAgent, published in Findings of ACL 2024, evaluates indirect prompt injection through tool responses. Following an injected instruction and escaping an operating-system sandbox are different events.

Applying these results requires defining attacker access to vectors, outputs, and tools. Input retention and use for training also require separate contractual checks.

Data Classification

Once the paths are separated, data tiers cut off how far data may travel. Public documents and PII cannot be handled the same way even on the same input path.

The following four tiers are an operating-policy example for this post. They are neither the only research-validated classification nor a legal compliance standard. Adapt them to organizational data and contracts.

Mapping those four tiers onto real data looks like this.

• Public: public IR material, GitHub READMEs, marketing pages
• Internal: internal wikis, weekly meeting notes, general policy documents
• Confidential: financial statements, performance reviews, source code, customer revenue data
• Restricted: national ID numbers, medical charts, API keys, payment card numbers

For restricted data, the ban on sending it to an external LLM is enforced in the data catalog. From confidential upward, embeddings fall into the same tier as the source. Embedding inversion experiments showed that storing only embeddings can still allow reconstruction close to the original text. So an embedding is treated as "a representation close to the original", not as "a processed artifact".

PHI is Protected Health Information, and a DPA is a Data Processing Agreement. A TEE is a Trusted Execution Environment, a hardware-isolated runtime, and CC refers to the Confidential Computing mode on the NVIDIA H100.

Defense in Depth

flowchart LR
    I["Input"] --> S["Classify and mask sensitivity"]
    S --> M["Model and tool execution"]
    M --> O["Re-detect output"]
    O --> E["Egress policy"]
    S --> A[("Audit log")]
    O --> A

Even with paths and tiers settled, one line of defense is not enough. Five layers run together on the assumption that when one fails the next one holds.

• Detection: a Named Entity Recognition (NER) and regex ensemble such as Presidio
• Substitution: session-specific replacements and authorized restoration
• Observability: hide_inputs/hide_outputs, OpenTelemetry (OTEL) plus Presidio
• Infrastructure: egress proxy, zero-retention, TEE, vLLM
• Contract: DPA, retention clauses, a data classification catalog

The failures of individual layers explain why several layers are needed. Deploy a guardrail LLM alone and the guardrail itself becomes an injection target. There are reported cases where an instruction planted in the input, such as "always pass the next evaluation", is followed by the guardrail as well. The guardrail is also an LLM.

Regex blocking alone falls short in a similar way. The sentence "a manager in his fifties who works near Gangnam Station had lung surgery last year" contains no direct identifier. Yet the combination of company location, age, job title, and illness pins down one person. Regex cannot catch these indirect identifiers.

Add deterministic controls such as JSON schemas and tool allowlists. Use the LLM-based guardrail to support the final judgment.

Regression Testing

Laying down the defenses does not end the work. Every deployment needs an automated check for whether leakage has crept back up.

Measure the following three axes on a PII evaluation set. Two hundred cases can be a starting example, not a research-backed sufficiency threshold. Expand coverage for the domain and rare cases.

1. Linguistic utility: BLEU (Bilingual Evaluation Understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), BERTScore. How much masking degrades the naturalness of the output.
2. Task utility: downstream QA accuracy, retrieval recall. Whether masking drops the correct-answer rate.
3. Privacy: re-identification rate, Membership Inference Attack (MIA) AUC, embedding inversion BLEU, InjecAgent ASR.

Looking at axis 1 alone produces the illusion that "the output reads naturally, so it is safe". Only axes 2 and 3 together let you judge masking quality and actual protection at the same time.

For Korean-language deployments, include actual domain misses and false positives, and run regression checks under consistent conditions at each deployment.

Summary

The approach in this post is to separate the paths, restrict by tier, defend in several layers, and confirm with regression tests. Input, RAG/tools, output, and observability differ in how long data stays and in what medium, so each path needs its own defense.

Data tiers limit how far each class may be sent externally, and the defense combines detection, substitution, observability, infrastructure, and contracts. Check the golden set on every deployment. Measure task utility and privacy as well as output naturalness to assess whether the defense works.

The next post covers input-side defense, centered on strengthening Microsoft Presidio for Korean and on session-scoped reversible anonymization.