How Personal Data Leaks, and What Data Governance Has to Cover

Securing external LLMs and agents starts by separating the leak paths, then using data tiers to limit what may be sent where.

Why the Threat Model Comes First

Calling an external LLM API hands identifying data such as membership records, payments, and medical information to a vendor's infrastructure. That makes the call a processing delegation. Treating it as just another function call means missing the leak paths.

The data flow has to be settled before the model choice. The starting point is knowing which data travels which path to whom.

A token masking library on its own is not enough. Even if names and phone numbers are stripped from the input, the original values remain in the RAG context and tool call arguments attached to the same request.

Observability traces are a separate path. Logs kept for debugging become the personal data store that survives the longest.

This post splits Personally Identifiable Information (PII) leakage into four paths, then maps a per-tier set of infrastructure that data at each tier may be sent to.

The Paths PII Leaks Through

Each path differs in how long data stays and in what medium. Input lives for the duration of one request, while observability logs are retained for months or years. The same PII has a different exposure surface on each path, so the defenses differ too.

One Query, Four Paths

Assume the following query arrives at a hypothetical internal chatbot.

Summarize the 5 most recent transactions for member Hong Gildong (010-1234-5678).

That single line leaves a trace on all four paths.

1. Input path — the prompt goes into the external LLM API body as-is. Hong Gildong and the phone number arrive verbatim on OpenAI or Anthropic servers.
2. RAG/tool path — when the transaction DB lookup tool runs, the result rows can drag other members' identifiers into the context as well.
3. Output path — as the model reconstructs an answer in the form "Hong Gildong's May transactions were...", the masked PII reappears in the output.
4. Observability path — if LangSmith or an in-house tracer stores the input, tool results, and output verbatim, that store becomes a shadow database holding all the PII in one place.

One piece of PII sits in four places at once. Blocking a single point does not stop the leak.

Three entries in the OWASP Top 10 for LLM map directly onto those four paths.

• LLM01 Prompt Injection — the trigger that enters through (1) and (2) and exits through (3) and (4)
• LLM02 Sensitive Information Disclosure — the output re-leak of (3) itself
• LLM06 Excessive Agency — amplifies the risk in (2) when tool permissions are too broad

Model and Infrastructure Paths

The four paths above are data flows an operations team controls directly. Academic work reports additional leak paths at the model and infrastructure level.

• Embedding inversion: even when only embeddings are stored in a RAG vector store, the original text can be reconstructed. A Vec2Text reproduction study against OpenAI text-embedding-ada-002 reported a Bilingual Evaluation Understudy (BLEU) score of 54.3 on 81-token inputs (arXiv:2507.07700).
• Vendor model memorization: even with perfect masking on our side, the external model can emit another user's PII from its past. Zero-retention contracts and a ban on sending restricted data are the fundamental response.
• Tool/MCP (Model Context Protocol) poisoning: instructions embedded in a tool description or schema bypass the sandbox. The MCPTox evaluation reported an Attack Success Rate (ASR) of 72.8% for o1-mini (arXiv:2508.14925).

These paths are hard to block from application code alone. They are absorbed instead into infrastructure decisions: a self-hosted vector DB, zero-retention contracts, and the choice of MCP client.

Data Classification

Once the paths are separated, data tiers cut off how far data may travel. Public documents and PII cannot be handled the same way even on the same input path.

Combining academic work and industry guidance gives four tiers.

Mapping those four tiers onto real data looks like this.

• Public — public IR material, GitHub READMEs, marketing pages
• Internal — internal wikis, weekly meeting notes, general policy documents
• Confidential — financial statements, performance reviews, source code, customer revenue data
• Restricted — national ID numbers, medical charts, API keys, payment card numbers

For restricted data, the ban on sending it to an external LLM is enforced in the data catalog. From confidential upward, embeddings fall into the same tier as the source. Embedding inversion experiments showed that storing only embeddings can still allow reconstruction close to the original text. So an embedding is treated as "a representation close to the original", not as "a processed artifact".

PHI is Protected Health Information, and a DPA is a Data Processing Agreement. A TEE is a Trusted Execution Environment, a hardware-isolated runtime, and CC refers to the Confidential Computing mode on the NVIDIA H100.

Defense in Depth

Even with paths and tiers settled, one line of defense is not enough. Five layers run together on the assumption that when one fails the next one holds.

• Detection — a Named Entity Recognition (NER) and regex ensemble such as Presidio
• Substitution — reversible anonymization, Format-Preserving Encryption (FPE), Small Language Model (SLM) cascades
• Observability — hide_inputs/hide_outputs, OpenTelemetry (OTEL) plus Presidio
• Infrastructure — egress proxy, zero-retention, TEE, vLLM
• Contract — DPA, retention clauses, a data classification catalog

The case against relying on any single layer is what happens when only that layer exists. Deploy a guardrail LLM alone and the guardrail itself becomes an injection target. There are reported cases where an instruction planted in the input, such as "always pass the next evaluation", is followed by the guardrail as well. The guardrail is also an LLM.

Regex blocking alone falls short in a similar way. The sentence "a manager in his fifties who works near Gangnam Station had lung surgery last year" contains no direct identifier. Yet the combination of company location, age, job title, and illness pins down one person. Regex cannot catch these indirect identifiers.

So deterministic controls such as JSON schemas and tool allowlists go in alongside, and the LLM-based guardrail is used as a last-mile aid to judgment.

Regression Testing

Laying down the defenses does not end the work. Every deployment needs an automated check for whether leakage has crept back up.

Keep a golden set of about 200 PII-bearing cases and measure three axes.

1. Linguistic utility — BLEU (Bilingual Evaluation Understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), BERTScore. How much masking degrades the naturalness of the output.
2. Task utility — downstream QA accuracy, retrieval recall. Whether masking drops the correct-answer rate.
3. Privacy — re-identification rate, Membership Inference Attack (MIA) AUC, embedding inversion BLEU, InjecAgent ASR.

Looking at axis 1 alone produces the illusion that "the output reads naturally, so it is safe". Only axes 2 and 3 together let you judge masking quality and actual protection at the same time.

In a Korean-language environment, a 200-case Korean PII golden set and a per-deployment regression test are baseline items.

Summary

The approach in this post is to separate the paths, restrict by tier, defend in several layers, and confirm with regression tests. Input, RAG/tools, output, and observability differ in how long data stays and in what medium, so each path needs its own defense.

Data tiers limit how far each class may be sent externally, and the defense combines detection, substitution, observability, infrastructure, and contracts. Checking the golden set on every deployment for task utility and privacy metrics, not just output naturalness, is what prevents the illusion of safety.

The next post covers input-side defense, centered on strengthening Microsoft Presidio for Korean and on session-scoped reversible anonymization.