Membership in a retrieval store and inclusion in model training are different events.
Membership inference estimates whether particular data belongs to a protected set. For RAG, that set can be the retrieval store, but training-data attack results do not transfer automatically.
This post uses established membership-inference research and RAG architecture to define evaluation questions. It does not present a particular RAG attack's high success rate or low query count as a universal service risk.
Training sets and retrieval stores
Shokri and colleagues' IEEE S&P 2017 study evaluates inference of training membership from model predictions. LiRA, published at IEEE S&P 2022, emphasizes evaluation at low false-positive rates.
The NeurIPS 2020 RAG study retrieves external documents for generation. This separates questions about training data from questions about the currently retrievable set. That structural distinction is not experimental evidence that the earlier attacks succeed against every RAG service.
| Question | Target set | State to check |
|---|---|---|
| Did the model train on this data? | Training data | Training date and model version |
| Is the document stored? | Complete document store | Storage and deletion state |
| Can this user retrieve it? | Authorized retrieval set | Requester, policy, and index state |
A stored document may be inaccessible to a requester. Its absence from search results also does not establish deletion from the source store.
Observable signals
The following table organizes a service threat model. It is neither a validated attack list nor a success-rate ranking from a particular paper.
| Exposed information | Risk to check | Required control condition |
|---|---|---|
| Document IDs and source text | Direct disclosure of membership and content | Document-level authorization |
| Citations and excerpts | Disclosure of restricted sources or content | Preserving permissions on material used in answers |
| Generated answers only | Whether answers reveal candidate-document information | Distinguishing generation errors and prior model knowledge |
| Scores and other metadata | Whether extra observations aid inference | Evaluation of each exposed field |
Similarity between an answer and a document does not establish store membership. Prior model knowledge, duplicate content, and generation errors can contribute. Evaluation needs controls with known membership.
Constructing a membership evaluation
The following flow proposes a local test. Compare member and non-member documents under the same model and retrieval settings, using only outputs available to the attacker.
flowchart LR
D[Documents with known membership] --> Q[Fixed queries and budget]
Q --> S[RAG service with authorization]
S --> O[Externally visible outputs]
O --> J[Membership decision]
J --> E[Evaluate true and false positives]Match controls reasonably on topic, length, and public availability. Otherwise, the attack may identify distribution differences instead of membership. Record query count and decision thresholds with the results.
| Measurement | Interpretation question |
|---|---|
| True-positive rate at low false-positive rate | Can it identify members while rarely misclassifying non-members? |
| Legitimate retrieval quality | Do controls obstruct retrieval of permitted documents? |
| Authorization violations | Are unauthorized documents retrieved or cited? |
| Query cost | How many requests are needed for one membership decision? |
Scope of access control
Consider enforcing document permissions before material reaches the model. Hiding content only in the final answer can leave unauthorized material in model context. This is a design judgment about data flow, not a measured blocking rate for a particular defense.
Authorization and output minimization also differ from formal privacy guarantees. Adding arbitrary noise does not establish differential privacy. Such a claim requires a protection unit, neighboring-dataset definition, mechanism, and budget.
LLM Privacy Defenses explains these guarantees. Changes to retrieval structure or exposed metadata require repeating the evaluation.
Summary
Training inclusion, storage, and user-specific retrievability are separate events. Established membership-inference studies provide evaluation concepts, not proof of an attack success rate against a particular RAG service. Service tests need controls with known membership and the information actually exposed. Evaluate authorization and legitimate retrieval quality together to establish the protection scope.