Egress Gateway Defense

Regex blocking at the gateway is not the first line of defense but the last safety net. It only means something when a contract that separately specifies request retention and training use, data-classification routing, and an immutable audit log (write once, no edits or deletes) sit above it.

Why external LLM traffic has to be funneled through one place

The larger a service gets, the more scattered its external LLM API call sites become. Agent workflows, a separate embedding pipeline, and ad-hoc scripts an operator wrote all go out with their own API keys and their own clients. With separate clients, you cannot trace which data goes to which model. During an incident, this delays the decision about which calls to block first.

An egress gateway routes these external calls through one place. Once all external LLM traffic passes through the same proxy, authentication, model routing, and key management can all be enforced from a single position. So can rate limiting, audit logging, and the final-stage PII regex block. During an incident, blocking that one gateway stops outbound transmission immediately.

The gateway's PII regex block is the last check in a cascade (a multi-stage defense applied in sequence). Input sanitization runs first, followed by masking at the tool and RAG stages. The gateway filters identifiers that remain after both. Without that framing, a regex that misses a piece of PII turns directly into a leak.

The five controls a gateway provides

flowchart LR
    A["Application"] --> G["Egress gateway"]
    G --> C["Data classification"]
    C --> R["Policy routing"]
    R --> P["Approved LLM provider"]
    G --> B["Block or mask"]
    G --> L[("Minimal audit log")]

In practice the industry-standard candidates for an egress gateway are LiteLLM Proxy, Cloudflare AI Gateway, and Portkey. The three differ in implementation but provide the following controls in common.

1. Model routing: selects an external model, a self-hosted vLLM, or a TEE according to the data classification level, enforcing the level-to-model mapping registered in the catalog.
2. API key abstraction: clients know only the gateway key; the gateway holds and rotates the real vendor keys. The blast radius of a key leak shrinks and revocation gets faster.
3. Rate limit + cost guardrail: enforces call-frequency and cost ceilings per user, team, and model. This prevents an infinite tool-call loop from turning into a billing disaster.
4. Audit log: records immutably who called which model, when, and with what input.
5. PII regex block: filters identifiers remaining in inputs and outputs at the final cascade stage.

The three gateways differ in where they run and how deeply they integrate. LiteLLM is a self-hostable Python proxy that exposes an OpenAI-compatible API across 100+ models. Callback hooks let you write input and output processing directly in code, which makes it easy to combine with your own masking and guardrails. Cloudflare AI Gateway is a managed gateway running at the edge, with caching, logging, and rate limiting as its strengths. Portkey offers multi-provider routing, fallback, and an analytics dashboard as a SaaS product.

The choice is usually made by compliance. Under Korea's PIPA (Personal Information Protection Act), routing confidential-or-higher data traffic through a processor's infrastructure may be uncomfortable. In that case, self-hosted LiteLLM is the safe answer. If most traffic is public or internal and a globally distributed cache is worth something, Cloudflare AI Gateway fits.

Retention and training-use policies

Request retention and use for training are separate conditions. The label zero-retention does not establish identical coverage for every endpoint, log, and exception, or automatically prohibit training use. Check the actual contract and the policies of enabled features.

Nasr and colleagues' ICLR 2025 study evaluates training-string extraction from aligned production models. Its preprint is arXiv:2311.17035. The study demonstrates training-data exposure in the tested models at that time, not inclusion of a particular customer's API requests in training.

The study also does not establish that a particular retention contract prevents leakage. Input minimization reduces what is currently sent; retention and training-use conditions govern provider handling. Output of previously learned material requires separate evaluation.

This table supports technical review, not a legal compliance determination. A particular region or contract is not presented as a necessary and sufficient condition for every service.

Data-classification routing

The following is a routing-policy example proposed in this series. Authenticated services assign data tiers and the gateway enforces policy. This is neither a product default nor the only research-validated policy.

The gateway rejects immediately when the level metadata is missing or the routing is wrong. The routing decision is a stronger control than regex blocking. A regex lets traffic through whenever it fails to match a pattern, whereas routing prevents the call from happening at all if the level does not permit it.

One item is frequently missed at this stage. Embeddings have to be part of the same classification system. Embedding inversion attacks reconstruct the source text from an embedding vector. Vec2Text, published at EMNLP 2023, evaluates text reconstruction with target vectors and the tested encoder access. Its recovery rates do not apply to arbitrary vector stores. Send embeddings of confidential-or-higher data to a self-hosted vector database, and cut off routing so the same data never reaches an external embedding API.

Regex blocking

Regex blocking at the final cascade stage is the position that catches what upstream masking missed. Expressed as a LiteLLM callback hook, it takes roughly this shape.

# pseudo-code
async def pre_call_hook(data, user_api_key_dict, call_type):
    text = extract_text(data)
    if has_pii_pattern(text):  # resident registration number, card number, account number, email, etc.
        raise BlockedRequest("egress PII pattern matched")
    log_audit(user_api_key_dict, data, redacted=False)
    return data

The flow is simple. Extract text from the request body and check it against the regex patterns. Reject the call if a pattern matches. The audit log is written unconditionally for every request.

Regexes catch only identifiers with an unambiguous format: resident registration numbers, card numbers, account numbers, emails, and phone numbers. Identifiers expressed in natural language and indirect identifiers (age + employer + medical condition) are beyond a regex. Input-stage contextual review and data minimization also need separate evaluation. Knowing that the gateway regex catches only clear patterns and blocks nothing else is what keeps you from a false sense of safety.

Audit logs

The greatest value a gateway creates is traceability during an incident. Immutable records identify who called which model, when, and with what input. These records let you determine the affected scope immediately after an incident.

• Append-only: a written record cannot be modified or deleted. A SHA-256 hash chain verifies integrity through prev_hash.
• WORM (Write Once Read Many) storage: S3 Object Lock in Compliance mode locks records against deletion for the retention period.
• Separation of privilege: separate the identity that writes the audit log from the identity that reads it, so an operator cannot erase their own call history.

There is a growing pattern of treating audit logs as evidence of "appropriate technical measures" under GDPR Art. 32 (Security of processing). They also serve as evidence of the strengthened safeguards required by Article 30 of the PIPA Enforcement Decree. The immutable log itself becomes a compliance deliverable.

Week one (Phase 0)

When first introducing a gateway, the following items can be finished within a week.

• Install an egress proxy such as LiteLLM or AI Gateway, plus an allowlist of external LLM endpoints (blocking calls to unregistered models)
• Distribute only gateway-scoped keys to clients; the gateway holds the vendor keys
• Enforce LangGraph interrupt() on every destructive tool
• LangSmith hide_inputs/hide_outputs + a Presidio callable, or self-hosted Langfuse
• Lock audit logs immediately with S3 Object Lock in Compliance mode

Expansion over the following month covers renewing zero-retention contracts, a four-level data classification catalog policy, and a 200-item PII golden-set regression test.

Summary

An egress gateway funnels external LLM traffic into one place and enforces model routing, key abstraction, rate limiting, audit logging, and PII regex blocking simultaneously. Choose among LiteLLM Proxy (self-hosted), Cloudflare AI Gateway (edge-managed), and Portkey (SaaS) according to compliance requirements.

Regex blocking is only the last safety net in the cascade. Request retention, training use, data-tier routing, and audit records require separate checks.

Embeddings belong in the same classification system. Calls whose level does not match are cut off by routing, not by a regex.

The final post covers output guardrails, observability, and the right to erasure, closing out the series.