The following scene is an illustration, not a customer case. In early March, a support lead approves a temporary exception. A billing migration double-charged a group of accounts, so for those accounts only, refunds requested after the usual 30-day window can be approved until March 31. The lead records the decision in a ticket, with the list of affected accounts attached and the end date in the ticket’s expiry field.
That week, a scheduled job summarizes recent policy changes into the knowledge base the support assistant reads. The summary says: Refunds requested after 30 days can be approved for accounts affected by the billing migration. The sentence is accurate. In May, a customer whose account was never affected asks about a late refund. The assistant retrieves the summary, finds a rule that matches the question, and tells the customer the refund can be approved.
An AI summary can repeat a rule accurately and still lose the conditions that decide when the rule applies. Who it covers, how long it is valid, which rule it overrides, and who approved it all belong to the data. When those conditions do not survive summarization, chunking, or transformation, the AI applies a correct sentence to the wrong case.
Key takeaways
- An AI summary can match its source word for word and still drop the scope, end date, precedence, and approver of a rule.
- Freshness checks miss this failure: the document is current, but the rule inside it has ended.
- Keep conditions as metadata fields on every chunk and summary, such as
scope,end_date,overridden_rule_id, andsource_record_id, and filter on them at retrieval. - Test the edges with fixed evaluation conditions: the day after expiry, an account outside the scope, and a question that retrieves both the exception and the rule it overrides.
Why an accurate AI summary can still give the wrong answer
An accurate AI summary can still give the wrong answer because summaries keep the main statement and drop the qualifiers, and the qualifiers are what limit a rule. In the source ticket, the conditions lived in three places:
- An attached account list that defined who the exception covered
- A structured date field that held the end date
- An approver line that recorded who authorized it
None of them were in the sentence the summarizer read most closely. The summary kept the rule and dropped the account list, the end date, and the approver.
- 01Source ticket
Rule, affected account list, end date of March 31, and approver
- 02AI summary
Rule only: refunds after 30 days can be approved for affected accounts
- 03Retrieval in May
The summary matches a late-refund question from an account that was never affected
- 04Answer
Refund approved for a case the exception never covered
Splitting documents for retrieval causes the same loss at a smaller scale. In its write-up on Contextual Retrieval, Anthropic describes a chunk from a financial filing that reports revenue growth without saying which company or which quarter it refers to. Their approach adds a short explanation of that context to each chunk before indexing. The same thing happens to an exception: once the paragraph is separated from the ticket, nothing in it says the rule ended on March 31.
The failure is hard to spot in review. Anyone who checks the summary against the ticket will find that it matches. The error appears only when someone asks a question the exception was never meant to answer.
Why stale data checks miss an expired exception
Data freshness checks would not have caught this failure, because the summary was not stale data. It was the newest entry in the knowledge base about refunds. Nobody had edited the ticket since March, and nothing about the document was out of date. The rule inside it had ended.
Re-indexing more often would not help either. A nightly refresh would summarize the same ticket again and produce the same sentence. A freshness check asks whether a document is the latest version. The question the support assistant needed answered was different: does this statement still apply, and to whom?
| Check | What it asks | What it catches | What it misses |
|---|---|---|---|
| Freshness check | Is this the latest version of the document? | Outdated versions and updates that never reached the index | A current document whose rule has already ended |
| Validity check | Does this statement still apply today, and to whom? | Rules past their end date and cases outside their scope | Changes to the document itself, so it does not replace the freshness check |
The two checks answer different questions, so one cannot stand in for the other. Keep the freshness check for document versions, and add a validity check for the rules inside them.
Conflicting information in enterprise RAG: which rule takes precedence?
When an enterprise RAG system retrieves both a general rule and an exception, the model needs to know which one takes precedence, and the text alone rarely says so. In the refund scene, a question about late refunds could retrieve the standard 30-day policy and the exception summary together. Without a link between them, the model has two plausible answers. It may pick one, or it may blend them into something neither document said, such as refunds are usually limited to 30 days but can often be approved later.
The two documents did not contradict each other. Both were correct, for different accounts and different dates. What the data lacked was the relationship: this exception overrides that rule, for this scope, until this date.
Five conditions an AI summary has to carry
Before a rule, policy, or decision enters a summary or a retrieval index, check that five conditions travel with it: scope, validity, precedence, authority, and source. Each one answers a question the AI will otherwise guess at.
| Condition | In the source (illustration) | What the summary kept | How to carry it | How to check it |
|---|---|---|---|---|
| Scopewho or what it covers | Accounts double-charged in the migration, listed in an attachment | Accounts affected by the migration, with no list | Keep the account list, or a linked rule_, in a scope field | Ask about an account outside the list; the answer should not apply the exception |
| Validitywhen it starts and ends | Until March 31, in the expiry field | Nothing | Store start_ and end_ as filterable fields and use them at retrieval | Ask after the end date; the answer should give the standard rule |
| Precedencewhich rule it overrides | Replaces the 30-day window for those accounts only | Reads as an alternative to the general policy | Link the exception to the rule it overrides with overridden_ | Ask a question that retrieves both; the answer should say which one applies |
| Authoritywho approved it | The support lead, named in the ticket | Nothing | Keep approver and decision_ as fields | The answer should be able to point to the approved record |
| Sourcewhere it came from | Ticket ID and policy version | A link to the knowledge base page | Keep source_ and the ingestion_ the summary was made from | When the source changes, regenerate the summary and rerun the boundary cases |
The fourth column matters as much as the third. A condition that the pipeline records but never tests can disappear again at the next change, and nobody will notice until a customer asks the wrong question.
Keep conditions in document metadata, not only in the summary text
The most reliable way to keep conditions through a RAG pipeline is to store them as document metadata that travels with every chunk and summary, rather than trusting the summary text to repeat them. The next summarization step can rewrite a sentence, but it leaves an end-date field as it is.
- Attach conditions as fields. Keep
scope,start_date,end_date,overridden_rule_id,approver, andsource_record_idwith each chunk, with its embedding, and with any summary made from it. - Use the fields at retrieval. Filter out or score down items whose
end_datehas passed before they reach the model, and passscopealong so the model can check whether the case in front of it qualifies. - Write the context into each chunk header. A short header such as
[Temporary exception | Valid: Mar 1 to Mar 31 | Scope: migration-affected accounts only]keeps the limits in front of the model even when a field is lost downstream. - Regenerate from the source, not from the last summary. When a summary is built from an earlier summary, each pass can drop another qualifier.
This is the same discipline that keeps semantic context attached to data for agents, and it overlaps with context engineering. Both depend on passing the model the limits it has to apply. The same fields help the person who approves the answer: when conditions arrive with the draft, a reviewer does not have to rebuild them by hand.
Test the conditions, not only the sentence
To know whether an AI applies an exception correctly, test the cases at its edges: the day after it ends, an account outside its scope, and a question that retrieves both the exception and the rule it overrides. A summary that passes a word-for-word comparison can still fail all three.
Run the boundary cases as a fixed routine:
- Fix the evaluation conditions. Use the same model or agent, the same prompt, and the same retrieval settings.
- Run the three boundary cases. The day after the exception ends, an account outside its scope, and a question that retrieves both the exception and the rule it overrides.
- Compare answers before and after the data change. With the setup fixed, the difference reflects the data, not a moving configuration.
- Treat thin evidence as inconclusive. If the cases are too few to tell, add cases before deciding.
- Rerun when the source changes. Every change to the source record triggers the boundary cases again.
The same habit applies after model performance drops.
Exception records are a practical place to start. A delivery team that records every exception with its scope, owner, and end date gives the next project something it can check. We covered that handoff in what a second AI deployment should inherit.
Where conditions fit in AI data preparation
Keeping conditions attached is part of preparing data for a specific use case, alongside field meanings, business terms, and relationships between records. In Syntitan, CUBIG’s AI-Ready Data Platform, teams prepare data for a defined use case, validate with the model or agent they selected, under fixed evaluation conditions, whether the data meets the criteria for that task, and review changes and revalidate when the underlying data changes. Whether data is AI-ready depends on the task it serves. An expired exception can make the same data unfit for that task without anyone editing a file.
Five questions to ask about your AI summaries
- When a summary states a rule, can we tell whom it covers and until when?
- If two retrieved documents disagree, does the data say which one takes precedence?
- Do our chunks and summaries carry
end_dateandsource_record_idas fields? - Have we tested the day after an exception ends, and an account outside its scope?
- When the source record changes, do we know which answers to check again?
If you cannot answer the first question for most of your knowledge base, start there. Pick one rule with an end date, follow it from the source to the assistant’s answer, and see which conditions arrive.
If your team prepares enterprise data for AI tasks and wants to check that the conditions survive, explore Syntitan.
