A relationship manager asks the bank’s AI assistant a specific question: can we waive the early repayment fee for this client? The assistant finds the right facility agreement. Before the text reaches the model, the names and the facility amount are replaced. The answer comes back polite and generic. It explains how waivers usually work and asks the manager to check the contract.
The waiver clause applied only when the borrower’s parent guaranteed the facility and the amount stayed under a threshold. Both facts were in the document. During substitution, the borrower and its parent company, which guaranteed the facility, became the same generic label, and the amounts became placeholders. The model could no longer tell who guaranteed whom or whether the amount met the waiver condition.
- 01Ask
Can we waive the early repayment fee for this client?
- 02Retrieve
The assistant finds the right facility agreement.
- 03Substitute
Borrower and parent become the same label; amounts become placeholders.
- 04Generic answer
The model cannot tell who guaranteed whom or whether the amount qualifies.
The document was available. The details needed to answer weren’t. The failure came from a substitution rule that erased the distinctions the task needed. Before you rely on substituted text, check that it still answers the questions your team asks.
Key takeaways
- Substitution can keep selected original values out of the model input. Teams still need to check whether the task works on the substituted version.
- Many tasks depend on details beyond the values themselves: which records refer to the same entity, how entities relate, whether an amount meets a threshold, and the dates the rule checks.
- Check the task, not only the substitution: run the questions your team asks on the protected copy and compare the answers with reference answers you already trust.
LLM data privacy: what teams need to check after substitution
Replacing sensitive values before a model call addresses one requirement: keeping selected original values out of the model input. Agent permission checks address another; in October, ServiceNow announced an AI Passport, expected in the fourth quarter of 2026, that verifies what an agent may do at each step.
Authorization checks and value substitution alone do not establish whether the substituted text supports a specific question. Teams still need to check that. Our earlier piece on where retrieved context leaks in RAG covers where sensitive context travels. This post covers what the task still needs once it is substituted.
What an AI task needs to keep after substitution
A substitution rule decides value by value: is this field sensitive? A task depends on how values connect. The table lists details that often carry the answer, a substitution that breaks each one, and what to check for instead. It is a checklist to run per task, not a list of features any one tool provides in every case.
| Detail the task depends on | Substitution that breaks it | What to check after substitution |
|---|---|---|
| Same entity across records | A new random stand-in in every document or session | The same entity gets the same stand-in across the workflow |
| Relationships between entities | Borrower, parent and guarantor all replaced with the same generic label | Different entities keep distinct stand-ins, so who relates to whom stays readable |
| Amounts against a rule | Every amount replaced with one placeholder | Whether the rule can still be checked, for example whether an amount meets a fixed waiver threshold |
| The dates and time gaps | Dates removed, so within 30 days cannot be checked | Whether the rule checks an interval or a fixed date, and that the substituted text still supports it |
| Internal codes and terms | Codes treated as sensitive text and dropped | Codes keep their business meaning |
Preserving a time interval or an ordering is not enough when the rule depends on a fixed date or amount, such as a policy start date, a maturity date or a waiver threshold. Keep the substitution, and decide what it must preserve per task, with the people who know the task. Details that matter but are not personal data, such as deal terms or internal codes, need the same attention; see confidential business context beyond PII.
The form of the stand-in can decide whether the task still works, and that can be measured. In a controlled benchmark CUBIG published on Hugging Face, one model (gpt-6-luna) extracted identifiers from 500 Korean documents under four conditions. Mean value-set extraction F1 was 98.0 on the original text and 0.0 when the values became ***. Hashed codes, mapped back afterward, scored 76.7. Stand-ins that keep each value’s type scored 99.3: the model returned the stand-ins, and a local table mapped them back to the original values. The benchmark tests a replacement principle used in LLM Capsule. It covers one model and two question types, it is not an end-to-end test of the product, and the slightly higher score does not mean substitution improves answers in general. The results and the dataset are public.
How to check that the task still works after substitution
You can run this check with the tools you already use. It needs a short list of real questions and answers your team already trusts.
- Pick the task and its real questions. Use questions people actually ask, such as fee waivers, claim eligibility or contract renewals.
- Mark the details each answer depends on. Note the entities, relationships, amounts and dates the right answer needs.
- Run the questions on the protected copy. Compare the answers with reference answers your team trusts, using the same criteria each time.
- Check the restored answer. Confirm that the real values came back to the right places and that a reviewer can see what was replaced.
- Record the rules you tested. Keep the substitution rules and the question set together, and run the check again when either changes.
Where LLM Capsule fits
Building context-preserving substitution yourself means keeping stand-ins consistent across documents and sessions, keeping relationships readable, and mapping every value back correctly in the answer. That is the work CUBIG built into LLM Capsule, a context-preserving data layer for AI.
LLM Capsule replaces controlled values with context-preserving stand-ins before they reach AI, then restores the original business values in the AI response. The originals and the mapping stay in your environment. LLM Capsule maps each stand-in back to its original value. Calculations based on substituted amounts or dates may not match calculations on the originals. Mapping stand-ins back to original values does not, by itself, correct those derived results. Restoring values does not make the AI’s judgment correct, and it does not make a workflow compliant on its own. Supported models, deployment location and restore scope are confirmed for each customer environment. For the term itself, see our glossary entry on context-preserving data layers.
Before your next AI rollout on sensitive data, take five questions your team asks every week and run them on the protected copy.