Egress Gateway Defense

Regex blocking at the gateway is not the first line of defense but the last safety net. It only means something when a zero-retention contract (a policy of not retaining or training on request data), data-classification routing, and an immutable audit log (write once, no edits or deletes) sit above it.

Why external LLM traffic has to be funneled through one place

The larger a service gets, the more scattered its external LLM API call sites become. Agent workflows, a separate embedding pipeline, and ad-hoc scripts an operator wrote all go out with their own API keys and their own clients. In that scattered state you cannot trace which data is going to which model, and when an incident happens the decision about where to start cutting off arrives late.

An egress gateway is the device that pulls that sprawl into one place. Once all external LLM traffic passes through the same proxy, authentication, model routing, and key management can all be enforced from a single position. So can rate limiting, audit logging, and the final-stage PII regex block. During an incident, blocking that one gateway stops outbound transmission immediately.

One point of framing matters here. The gateway's PII regex block is not the first line of defense but the last safety net in a cascade (a multi-stage defense stacked in series). The first line is input sanitization, the second is masking at the tool and RAG stages, and the gateway is the position that filters whatever residue got through both. Without that framing, a regex that misses a piece of PII turns directly into a leak.

The five controls a gateway provides

In practice the industry-standard candidates for an egress gateway are LiteLLM Proxy, Cloudflare AI Gateway, and Portkey. The three differ in implementation but provide the following controls in common.

1. Model routing — selects an external model, a self-hosted vLLM, or a TEE according to the data classification level, enforcing the level-to-model mapping registered in the catalog.
2. API key abstraction — clients know only the gateway key; the gateway holds and rotates the real vendor keys. The blast radius of a key leak shrinks and revocation gets faster.
3. Rate limit + cost guardrail — enforces call-frequency and cost ceilings per user, team, and model. This prevents an infinite tool-call loop from turning into a billing disaster.
4. Audit log — records immutably who called which model, when, and with what input.
5. PII regex block — filters identifiers remaining in inputs and outputs at the final cascade stage.

The three gateways differ in where they run and how deeply they integrate. LiteLLM is a self-hostable Python proxy that exposes an OpenAI-compatible API across 100+ models. Callback hooks let you write input and output processing directly in code, which makes it easy to combine with your own masking and guardrails. Cloudflare AI Gateway is a managed gateway running at the edge, with caching, logging, and rate limiting as its strengths. Portkey offers multi-provider routing, fallback, and an analytics dashboard as a SaaS product.

The choice is usually made by compliance. Under Korea's PIPA (Personal Information Protection Act), routing confidential-or-higher data traffic through a processor's infrastructure may be uncomfortable. In that case, self-hosted LiteLLM is the safe answer. If most traffic is public or internal and a globally distributed cache is worth something, Cloudflare AI Gateway fits.

Zero-retention contracts

Before the gateway sends anything, the first thing to confirm is the vendor's retention policy. The contract must lock in a zero-retention option so the processor does not use your requests for training or model improvement.

This control is more fundamental than masking because of research into model memorization, the phenomenon of a model retaining training data verbatim. Nasr 2023 (Scalable Extraction of Training Data from Production Language Models, arXiv:2311.17035) extracted training data verbatim from ChatGPT (gpt-3.5-turbo). Roughly 200 dollars' worth of queries produced more than 10,000 unique training examples, and the authors reported that spending more would yield more. The implication is that even with perfect masking on your side, the vendor's model may be holding another user's PII from the past and mix it into your response. That leak is not one you can block.

From a legal and compliance standpoint, the following three-part set is the minimum.

• PIPA: consent and notification for cross-border transfer, disclosure of processing delegation, retention and destruction policy. For a cloud LLM, the minimum set is zero-retention terms + a Korean region + a DPA (Data Processing Agreement).
• GDPR: requires SCCs (Standard Contractual Clauses) + a TIA (Transfer Impact Assessment).
• HIPAA: restricted to vendors that will sign a BAA (Business Associate Agreement), such as AWS Bedrock and Azure OpenAI.

Only on top of these contracts do masking and regex blocking mean anything. Enforce a catalog policy that restricted data is never sent to a model without a zero-retention contract, at any level of masking.

Data-classification routing

Every request the gateway receives has to carry a data classification level as metadata. Routing by level is enforced as follows.

The gateway rejects immediately when the level metadata is missing or the routing is wrong. The routing decision is a stronger control than regex blocking. A regex lets traffic through whenever it fails to match a pattern, whereas routing prevents the call from happening at all if the level does not permit it.

One item is frequently missed at this stage. Embeddings have to be part of the same classification system. Embedding inversion attacks reconstruct the source text from an embedding vector. Research on them has accumulated to the point where storing only embeddings can, under certain settings, still yield something close to the original (Vec2Text, Morris 2023). Send embeddings of confidential-or-higher data to a self-hosted vector database, and cut off routing so the same data never reaches an external embedding API.

Regex blocking

Regex blocking at the final cascade stage is the position that catches what upstream masking missed. Expressed as a LiteLLM callback hook, it takes roughly this shape.

# pseudo-code
async def pre_call_hook(data, user_api_key_dict, call_type):
    text = extract_text(data)
    if has_pii_pattern(text):  # resident registration number, card number, account number, email, etc.
        raise BlockedRequest("egress PII pattern matched")
    log_audit(user_api_key_dict, data, redacted=False)
    return data

The flow is simple. Pull the text out of the request body, run regex pattern matching over it, and reject the call outright on a match. The audit log is written unconditionally for every request.

Regexes catch only identifiers with an unambiguous format: resident registration numbers, card numbers, account numbers, emails, and phone numbers. Identifiers expressed in natural language and indirect identifiers (age + employer + medical condition) are beyond a regex. That responsibility belongs to the SLM cascade at the input stage. Knowing that the gateway regex catches only clear patterns and blocks nothing else is what keeps you from a false sense of safety.

Audit logs

The greatest value a gateway creates is traceability during an incident. Recording who called which model, when, and with what input, immutably, is what makes it possible to scope the blast radius right after an incident.

• Append-only — a written record cannot be modified or deleted. A SHA-256 hash chain verifies integrity through prev_hash.
• WORM (Write Once Read Many) storage — S3 Object Lock in Compliance mode locks records against deletion for the retention period.
• Separation of privilege — separate the identity that writes the audit log from the identity that reads it, so an operator cannot erase their own call history.

There is a growing pattern of treating audit logs as evidence of "appropriate technical measures" under GDPR Art. 32 (Security of processing). They also serve as evidence of the strengthened safeguards required by Article 30 of the PIPA Enforcement Decree. The immutable log itself becomes a compliance deliverable.

Week one (Phase 0)

When first introducing a gateway, the following items can be finished within a week.

• Install an egress proxy such as LiteLLM or AI Gateway, plus an allowlist of external LLM endpoints (blocking calls to unregistered models)
• Distribute only gateway-scoped keys to clients; the gateway holds the vendor keys
• Enforce LangGraph interrupt() on every destructive tool
• LangSmith hide_inputs/hide_outputs + a Presidio callable, or self-hosted Langfuse
• Lock audit logs immediately with S3 Object Lock in Compliance mode

Expansion over the following month covers renewing zero-retention contracts, a four-level data classification catalog policy, and a 200-item PII golden-set regression test.

Summary

An egress gateway funnels external LLM traffic into one place and enforces model routing, key abstraction, rate limiting, audit logging, and PII regex blocking simultaneously. Choose among LiteLLM Proxy (self-hosted), Cloudflare AI Gateway (edge-managed), and Portkey (SaaS) according to compliance requirements.

Regex blocking is only the last safety net in the cascade. It means something only when a zero-retention contract, data-level routing, and an immutable audit log sit alongside it.

Embeddings belong in the same classification system. Calls whose level does not match are cut off by routing, not by a regex.

The final post covers output guardrails, observability, and the right to erasure, closing out the series.