An operations analyst pastes a customer complaint export into an external chatbot to summarize recurring issues. The export contains customer names, account numbers and contract terms, so the chatbot service now holds information the company never intended to share with it. CUBIG checks which values may reach an AI service as part of checking whether data is ready for an AI task.
AI data leakage happens in two directions. A team sends information into an AI model or service that should not receive it, or an AI application returns sensitive information in its output to someone who should not see it.
NIST lists data privacy among the risks of generative AI in its July 2024 Generative Artificial Intelligence Profile (NIST AI 600-1). NIST describes the risk as impacts due to the leakage and unauthorized use, disclosure or de-anonymization of personally identifiable information or sensitive data, including biometric, health and location information.
OWASP covers the output side of the problem in LLM02:2025 Sensitive Information Disclosure, part of its 2025 Top 10 for LLM Applications. OWASP names personal information, financial details, health records, confidential business data, security credentials and legal documents as sensitive information. OWASP warns that LLMs embedded in applications can expose such data through their output, and that system prompt restrictions may not always be honored and can be bypassed through prompt injection or other methods.
AI data leakage can follow four paths through an enterprise AI workflow:
- Prompts and pasted text: an employee sends customer records, internal discussions or financial details directly to an AI model.
- Uploaded files and retrieved documents: a retrieval (RAG) application passes sensitive passages to a model that should not receive them, a case CUBIG covers in its article on RAG context leakage.
- Agent tool calls and tool outputs: an AI agent reads data through a connected tool and passes sensitive values on to a model or another service, as described in CUBIG’s article on agent tool output leakage.
- Model outputs: an AI application repeats names, account numbers, credentials or confidential details in a response to someone who should not see them.
How is AI data leakage different from data leakage in machine learning?
AI data leakage and data leakage in machine learning share a word but describe different problems.
- Data leakage in machine learning occurs when information that would not be available at prediction time, or that comes from the evaluation data, enters model training and inflates evaluation results.
- AI data leakage occurs when sensitive or confidential information is sent into an AI system, or returned by one, to a person, model or service that should not have it.
- A team investigating the first problem examines which fields, records and preparation steps carried information into training that the model would not have at prediction time. A team investigating AI data leakage examines where sensitive values travel and who or what can receive them.
How do teams reduce AI data leakage without blocking AI work?
The data team finds and classifies sensitive values before enterprise data leaves its boundary. Personal information, financial details, credentials and confidential business terms are easier to handle when the team knows where they sit before anyone sends a file to an AI service.
The platform team limits what each user, model and agent can reach. OWASP recommends strict access controls based on least privilege and restricting the external data sources a model can access, so an agent reads only the information its task needs.
The project owner decides which values the AI task actually needs. A complaint summary needs the context of the complaint, but it may not need the customer’s account number. Sending only the fields the task uses reduces what can leak.
When a task needs the context of sensitive values but not the values themselves, LLM Capsule is one approach CUBIG offers. LLM Capsule replaces sensitive or controlled values with context-preserving substitutes before data is sent to an AI service, then restores the original values in the response so the result can be used in the actual work. LLM Capsule can be adopted on its own.
CUBIG Shorts
Where LLM Capsule Sits in Your Workflow
A short clip from CUBIG on The Ravit Show.
The AI readiness checklist covers checking usage rights and identifying sensitive values before an AI project starts.