What is AI Data Leakage?

An operations analyst pastes a customer complaint export into an external chatbot to summarize recurring issues. The export contains customer names, account numbers and contract terms, so the chatbot service now holds information the company never intended to share with it. CUBIG checks which values may reach an AI service as part of checking whether data is ready for an AI task.

AI data leakage happens in two directions. A team sends information into an AI model or service that should not receive it, or an AI application returns sensitive information in its output to someone who should not see it.

NIST lists data privacy among the risks of generative AI in its July 2024 Generative Artificial Intelligence Profile (NIST AI 600-1). NIST describes the risk as impacts due to the leakage and unauthorized use, disclosure or de-anonymization of personally identifiable information or sensitive data, including biometric, health and location information.

OWASP covers the output side of the problem in LLM02:2025 Sensitive Information Disclosure, part of its 2025 Top 10 for LLM Applications. OWASP names personal information, financial details, health records, confidential business data, security credentials and legal documents as sensitive information. OWASP warns that LLMs embedded in applications can expose such data through their output, and that system prompt restrictions may not always be honored and can be bypassed through prompt injection or other methods.

AI data leakage can follow four paths through an enterprise AI workflow:

  • Prompts and pasted text: an employee sends customer records, internal discussions or financial details directly to an AI model.
  • Uploaded files and retrieved documents: a retrieval (RAG) application passes sensitive passages to a model that should not receive them, a case CUBIG covers in its article on RAG context leakage.
  • Agent tool calls and tool outputs: an AI agent reads data through a connected tool and passes sensitive values on to a model or another service, as described in CUBIG’s article on agent tool output leakage.
  • Model outputs: an AI application repeats names, account numbers, credentials or confidential details in a response to someone who should not see them.

How is AI data leakage different from data leakage in machine learning?

AI data leakage and data leakage in machine learning share a word but describe different problems.

  • Data leakage in machine learning occurs when information that would not be available at prediction time, or that comes from the evaluation data, enters model training and inflates evaluation results.
  • AI data leakage occurs when sensitive or confidential information is sent into an AI system, or returned by one, to a person, model or service that should not have it.
  • A team investigating the first problem examines which fields, records and preparation steps carried information into training that the model would not have at prediction time. A team investigating AI data leakage examines where sensitive values travel and who or what can receive them.

How do teams reduce AI data leakage without blocking AI work?

The data team finds and classifies sensitive values before enterprise data leaves its boundary. Personal information, financial details, credentials and confidential business terms are easier to handle when the team knows where they sit before anyone sends a file to an AI service.

The platform team limits what each user, model and agent can reach. OWASP recommends strict access controls based on least privilege and restricting the external data sources a model can access, so an agent reads only the information its task needs.

The project owner decides which values the AI task actually needs. A complaint summary needs the context of the complaint, but it may not need the customer’s account number. Sending only the fields the task uses reduces what can leak.

When a task needs the context of sensitive values but not the values themselves, LLM Capsule is one approach CUBIG offers. LLM Capsule replaces sensitive or controlled values with context-preserving substitutes before data is sent to an AI service, then restores the original values in the response so the result can be used in the actual work. LLM Capsule can be adopted on its own.

CUBIG Shorts

Where LLM Capsule Sits in Your Workflow

A short clip from CUBIG on The Ravit Show.

▶  Watch on YouTubeSubscribe

The AI readiness checklist covers checking usage rights and identifying sensitive values before an AI project starts.

Frequently asked questions

What is an example of AI data leakage?

An employee pastes customer records containing names and account numbers into an external AI chatbot to get a summary. An AI application that returns confidential details to a user who should not see them is another example.

Is AI data leakage the same as OWASP's sensitive information disclosure?

The two overlap. OWASP's LLM02:2025 Sensitive Information Disclosure focuses on sensitive data an LLM application exposes, while AI data leakage also covers sensitive data a team sends into an AI service.

How can companies reduce AI data leakage?

Companies can classify sensitive values before data leaves its boundary, restrict access by least privilege and send each AI task only the fields it needs. Some teams also replace sensitive values with context-preserving substitutes before the data reaches an AI service and restore the original values in the response, which is the approach LLM Capsule takes.