Artificial Intelligence

RAG Permission Revocation: Testing Stale Access Paths

Documents passing through an access check before reaching a retrieval system
Pew Nguyen

RAG permission revocation is not complete when a source document’s access list changes. Test the entire answer path: the identity supplied to retrieval, the permissions stored in the index, any live verification, and any application cache that can bypass retrieval. Bedrock Managed Knowledge Base filters against supplied user context but does not authenticate the end user. If an application supplies the wrong identity, retrieval filtering cannot repair that mistake.

This is a revocation test plan, not a general guide to building a managed RAG index. Its central question is practical: after access is removed at the source, can the former reader still obtain a document excerpt, citation or answer? The answer depends on the connector, synchronization state and application design. Treat each path as a separate boundary rather than declaring the system secure because a search query contains a permission filter.

Trace the answer path

Draw the path from the authenticated caller to the final response before changing configuration. Record where the application derives user identity, where it resolves groups, which index fields hold permissions, and whether the retriever checks the source again. Then add every place an earlier answer or retrieved passage can survive: response caches, conversation history, stored citations and application-side search results. A retrieval check does not retroactively remove text already copied into a conversation or cached answer.

Azure AI Search security filters compare an identity string supplied by the application with indexed document data. That makes the application responsible for deriving the string from a trusted session and applying the filter on every relevant query. Microsoft also describes token-based document permission approaches, but several are marked as preview; do not treat a string filter and a token-validated permission filter as interchangeable security mechanisms. An index credential that lets the application read documents is not, by itself, proof that a particular end user may read them.

For this audit, make an inventory with columns for component, identity source, permission source, refresh trigger, failure behavior and owner. Include the retrieval call and the answer cache as different rows. If the application has a fallback search path, list it too. This inventory is an original worksheet for finding bypasses; it is not a claim that either provider supplies these controls automatically.

Separate sync from live checks

Bedrock documents real-time ACL verification for supported connectors, while S3 and Custom connectors have no real-time verification. Its connector documentation describes a pre-retrieval stage using permissions collected during ingestion and a live source check for supported connectors. The distinction matters when a permission changes between crawls: a result filtered using indexed metadata alone can reflect an older access state. Confirm the behavior of the connector actually deployed, not that of another connector in the same service.

Azure AI Search enforces token-based document permissions against metadata already synchronized into the index. A source-side revocation therefore needs the documented synchronization mechanism to update the indexed permissions before that change is reflected in search results. The method varies by source and ingestion path. Preview features also carry deployment limitations, so an architecture review should record the exact approach and its support status rather than saying only “Azure ACLs enabled.”

Revocation pathWhat to testConservative application response
Indexed permissions awaiting refreshSearch immediately after source revocation, then after permission syncDo not rely on an unmeasured sync delay for sensitive content
Connector with live verificationRepeat retrieval after revocation without waiting for a crawlCheck how verification errors and unavailable sources affect results
Application response cacheRepeat the same question as the revoked user with a warm cacheInvalidate affected entries or reauthorize before serving them
Stored conversationReopen a prior answer containing the documentDefine access rules for saved output separately from retrieval

The table proposes tests and application responses; it does not report measured provider latency or guarantee instantaneous revocation. In particular, a live connector check does not validate a cached answer that never reaches the connector.

Run a controlled revocation test

Use a synthetic document with a unique, non-sensitive phrase and a test user whose access can be changed safely. Keep the phrase out of the question so its appearance in an answer or excerpt identifies a path from the document. Run the procedure in a controlled environment, with a test group if group permissions are part of the real design:

  1. Give the test user access and verify that retrieval finds the synthetic document. Record the authenticated principal, the permission representation in the index and whether the answer was served from a cache.
  2. Remove that user’s access at the source. Record the change event and keep the document itself unchanged, so a later difference cannot be attributed to reindexing its text.
  3. Query with the same authenticated user and a cold application cache before the next permission sync. Capture document identifiers and authorization outcomes, not sensitive passages.
  4. Repeat after the relevant permission sync or live-verification event. Test both a direct search result and the generated answer; a safe answer must not quote a forbidden passage merely because citation display was filtered.
  5. Repeat with a warm response cache and an existing conversation. These paths may not call retrieval at all and need their own invalidation or authorization policy.
  6. Introduce a controlled identity or verification failure. Confirm that the application does not silently retry without a permission constraint or switch to a broader search path.

Use explicit acceptance criteria: the revoked principal must not receive the synthetic document through fresh retrieval, a cached response or a saved view beyond the access policy you have defined. Record any interval during which indexed permissions remain stale, but do not publish it as a provider-wide guarantee. Bedrock documents fail-closed behavior for errors in its ACL evaluation pipeline; that statement does not establish how an application’s separate cache or fallback behaves.

Make the fix durable

Start at the earliest broken boundary. If identity is caller-supplied without trusted authentication, fix that before tuning an index. If the source change does not reach indexed permission metadata promptly enough for the workload, choose a supported live verification path or add an application-side source authorization check. If a cache serves old answers, key and invalidate it according to the application’s authorization model; provider-side prompt caching cost behavior is a different concern from an application cache of answers.

Keep permission metadata changes in the data-pipeline review. A full embedding reindex runbook addresses vector migration, but a permission-only change still needs its own propagation and verification test. Do not wait for an unrelated embedding migration to repair stale access. Restrict test logs to identifiers, decisions and timestamps needed for diagnosis; the synthetic phrase should not become a new sensitive payload in observability systems.

The method here is a proposed test, not a measured benchmark. It does not establish a universal revocation time, cover every connector configuration or resolve the policy for previously disclosed answers. Document those limits alongside the tested connector, identity model and cache behavior so another engineer can reproduce the result after a configuration change.

Sources

Keep exploring