Agent failure labs / Guide 66
A Highly Relevant Policy Can Still Be the Wrong Version
Reproduce a RAG outdated policy documents failure, then filter by effective date, tenant, and publication status before ranking evidence for a cited answer.
Illustrative engineering guide. Examples are not client deployments, production-ready integrations, or evidence of measured results. Validate the design with your domain and security owners.
“What meal allowance applied to my trip on September third?” A hypothetical employee asks after a policy update. The retriever returns an older document with nearly identical wording and a higher similarity score than the replacement. The answer cites a real document and quotes it accurately. It is still wrong for the date in the question.
This lab isolates policy eligibility before ranking. It is narrower than a general enterprise knowledge system: one policy family, one tenant, one effective date, and competing published versions. The exercise shows why adding citations does not fix temporal mistakes. Citation existence, relevance, authority, and applicability are separate properties.
Treat Time as an Eligibility Constraint
The applicable date is not always today. A historical expense, an employment event, or a contract renewal may require the policy effective when the event occurred. Ask for the relevant date if it is ambiguous. Store both the document publication timestamp and its effective interval; publishing a replacement early does not necessarily make it applicable immediately.
Azure AI Search’s filter documentation describes value-based filtering on nonvector fields, including metadata associated with chunks. Such filters can narrow the evidence set independently of relevance scoring. The implementation still has to choose the correct fields and semantics. A filter over inaccurate metadata makes the wrong answer look systematic rather than accidental.
Use a half-open interval: effective start is included, effective end is excluded. Python’s date documentation supplies date parsing and comparison for this local example. The interval convention below is an explicit application design choice, not a rule imposed by Python or a legal interpretation of policy wording.
from datetime import date
from copy import deepcopy
docs = [
{"id": "meal-v1", "tenant": "A", "family": "meals", "published": True,
"start": "2026-01-01", "end": "2026-09-01", "score": 0.99, "cap": 80},
{"id": "meal-v2", "tenant": "A", "family": "meals", "published": True,
"start": "2026-09-01", "end": None, "score": 0.91, "cap": 60},
{"id": "meal-draft", "tenant": "A", "family": "meals", "published": False,
"start": "2026-09-01", "end": None, "score": 0.98, "cap": 100},
]
query = {"tenant": "A", "family": "meals", "as_of": "2026-09-03"}
def naive(q, rows):
return max(rows, key=lambda d: d["score"])["id"]
def guarded(q, rows):
day = date.fromisoformat(q["as_of"])
eligible = [d for d in rows if d["tenant"] == q["tenant"]
and d["family"] == q["family"] and d["published"] is True
and date.fromisoformat(d["start"]) <= day
and (d["end"] is None or day < date.fromisoformat(d["end"]))]
if len(eligible) != 1:
return "abstain"
return eligible[0]["id"]
before = deepcopy(docs)
assert naive(query, docs) == "meal-v1"
assert guarded(query, docs) == "meal-v2"
assert guarded(dict(query, as_of="2026-08-31"), docs) == "meal-v1"
assert guarded(dict(query, as_of="2026-09-01"), docs) == "meal-v2"
assert guarded(dict(query, tenant="B"), docs) == "abstain"
overlap = docs + [dict(docs[1], id="conflicting-release")]
assert guarded(query, overlap) == "abstain"
assert guarded(query, docs[:1]) == "abstain"
assert docs == beforeThis is an explicitly synthetic deterministic simulation of a possible retrieval failure, not a real model experiment or search benchmark. Scores and allowance values are invented fixture data. The code exercises policy-version eligibility, not embeddings, language understanding, access-token verification, or answer generation. It intentionally abstains when two published versions overlap instead of choosing the higher score.
Why Prompt-Only Freshness Fails
“Use the latest policy” is the wrong instruction for a historical question and insufficient for a current one. The model may not see the effective date, may confuse upload time with publication time, or may receive only a chunk detached from its parent document. Attach authoritative version metadata to every retrieved chunk and preserve its relationship to the policy registry.
Version eligibility belongs before content ranking and must be checked again before answer release when policy state can change during generation. For one eligible version, rank its chunks to find relevant sections. Do not compare an obsolete chunk against an eligible one and hope the right section wins. If the eligible version lacks the needed clause, report that coverage gap rather than fall back to a superseded document.
Tenant and reader permissions are also eligibility constraints. OWASP’s authorization guidance recommends server-side permission checks on every request. Build security filters from authenticated identity, not text in the question. A document claiming “public to everyone” cannot grant access to itself. A retrieved instruction to ignore the policy registry is equally untrusted.
Make the Registry Authoritative
The minimum architecture consists of a policy registry, versioned document storage, an index carrying registry metadata, and an answer boundary that validates the selected version. A human policy owner approves publication and effective intervals. Content ingestion does not get permission to declare a policy active merely because a file contains a recent date.
Preserve stable policy-family identifiers through renames. Otherwise “Travel Meals” and “Meal Reimbursement” can become separate families with contradictory active rules. Handle amendments explicitly: either publish a consolidated version or model approved precedence. The toy single-version rule does not support layered jurisdictional exceptions, contractual overrides, or retroactive corrections without additional reviewed metadata.
The snippet omits authentication sources, database transactions, provider semantics, publication approvals, and durable state persistence. Updating a registry and its search index is not automatically atomic. A useful approach is an index generation associated with a registry release, plus a final authoritative eligibility check. If they disagree, abstain and expose the indexing delay to an operator.
Answer caches need policy release identity, tenant, permission scope, and applicable date in their cache key or invalidation rules. A corrected index is not enough if the application still serves yesterday’s generated answer. Store cited document version and section identifiers with each answer so a revoked release can be located and invalidated. Do not store private policy text in unscoped debug logs.
Retrieval timeouts mean unavailable evidence, not permission to answer from memory. If a downstream workflow would act on an answer, it must separately verify authorization, current preconditions, approval, and side-effect status. This lab remains read-only. It cannot produce a trustworthy action receipt, and a connector’s status: success would not prove that a policy was applied correctly.
Acceptance Tests
- The high-scoring obsolete document must win in the naive baseline and lose to the eligible version in the fixed path.
- The day immediately before replacement must select the previous version; the replacement date must select the new one.
- Draft documents must never become answer evidence simply because their text closely matches the question.
- Overlapping published intervals must produce abstention and an actionable registry-conflict report.
- A missing eligible document must remain an evidence gap rather than trigger fallback to an expired version.
- Permission changes and policy revocations must invalidate affected cached answers before they can be served again.
Measure Applicability Separately
Build a labeled evaluation set around release boundaries, historical dates, renamed families, and missing index generations. Measure applicable-version selections divided by questions with one authoritative applicable version. Report abstention precision separately: how many abstentions correspond to actual gaps or conflicts. Keep unauthorized queries and ambiguous-date requests in their own reported categories rather than hiding them as retrieval failures.
Use similarity-only retrieval as the baseline, then run the eligibility gate in shadow mode without changing staff answers. Proposed rollout criteria include correct date-boundary behavior and traceable release identity on every served answer. No production accuracy improvement is claimed. Human reviewers should inspect both confident answers and abstentions, because unnecessary blocking can conceal a damaged index.
The enterprise knowledge guide covers federated evidence access, while the expense enforcement lab shows where a dated allowance becomes an aggregate rule. AI systems engineering should begin with the policy registry and its owner. A better embedding model cannot decide which document was authorized to govern a particular day.