Training-data extraction, membership inference, and embedding inversion differ in access requirements and reconstruction targets.
Removing personal information from a prompt reduces the information sent in that request. It does not evaluate risks from previously trained data, stored vectors, or retrieval results.
This post distinguishes three attacks using representative research published at USENIX Security, IEEE S&P, EMNLP, and ICLR. Experimental results are not universal leakage rates for current services.
Access and attack targets
| Attack | Target information | Example observation required | Meaning of success |
|---|---|---|---|
| Training-data extraction | Specific strings included in training | Generated model responses | Recovering a candidate string and verifying its origin |
| Membership inference | Whether a sample was included in training | Model outputs, probabilities, or losses | Distinguishing members from non-members |
| Embedding inversion | The source text of a vector | A target vector and encoder access required by the attack | Recovering text or information about it |
Membership inference does not require full text reconstruction. Model extraction aims to reproduce model functionality or parameters; it does not by itself establish recovery of personal information.
Evidence for training-data extraction
Carlini and colleagues' USENIX Security 2021 study extracted and verified training strings from GPT-2. It establishes that a generative model can reproduce training data. It does not establish that every generated string is a training-data copy.
Nasr and colleagues' ICLR 2025 study extended extraction evaluation to aligned production models. Results from the tested models and interfaces at that time are not current API leakage rates.
flowchart LR
Q[Attack queries] --> M[Model]
M --> C[Generated candidate strings]
C --> V[Verify source text and inclusion]
V --> R[Verified extraction results]A candidate that resembles personal information is not necessarily recovered training data. Extraction evaluation needs source comparison and provenance checks. Provider retention periods and training-use policies require separate verification.
Membership decisions
LiRA, published at IEEE S&P 2022, studies membership inference. It emphasizes detecting actual members at low false-positive rates, rather than relying on average accuracy alone.
An attack threshold can incorrectly classify data that was not included in training. A single sample's high or low loss therefore cannot establish membership with certainty.
| Metric | Question |
|---|---|
| True-positive rate | What fraction of actual members did the attack identify? |
| False-positive rate | What fraction of non-members did it classify as members? |
| Attacker access | Can the attacker use probabilities, losses, or weights? |
| Comparison groups | Are member and non-member distributions appropriately matched? |
LiRA's training-data experiments do not validate the same attack against a particular RAG store. Membership in a retrieval store and membership in model training are separate events.
Conditions for embedding inversion
Vec2Text, published at EMNLP 2023, evaluates reconstructing text from embeddings. It reports 92% exact recovery under its tested embedding-model conditions for 32-token inputs. This rate does not apply to arbitrary lengths, models, or services.
Access to target vectors, available encoders, and the source-text distribution matter. A vector should not be treated as anonymous merely because its numbers are difficult to read.
| Protection point | Exposure reduced | Separate remaining issue |
|---|---|---|
| Input minimization | Personal information in requests | Previously trained data |
| Vector access control | Routes to raw embeddings | Information in permitted retrieval responses |
| Output inspection | Direct exposure of known sensitive patterns | Context, transformations, and missed detections |
| Differential privacy during training | Output-distribution differences between defined neighboring datasets | Raw stores and runtime service authorization |
Summary
Representative studies demonstrate training-data extraction, membership inference, and embedding inversion under different access conditions. One study's success rate is not another model's or RAG store's leakage rate. Service evaluation must first define what the attacker can observe and what counts as success. Input minimization, access control, and training-time protection address different parts of the problem.