MMOOZZGG.com

Glossary

What is AI data leakage?

In short

AI data leakage is a case in which company or customer data reaches outsiders because of a neural network: through employees’ prompts, model answers, or connected systems.

Security and law · data leakage

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you told a friend a secret, and the friend wrote it in a shared notebook that the whole class reads. Your friend did not mean any harm, they just did not think. A computer helper is sometimes like that notebook: what you write in it, someone else may see. So adults agree on which secrets must not be told to it, and they watch who can read its notes.

In plain words

Most often it happens with no hacking at all. An employee copies a contract, a customer table, or code into a public chatbot to finish work faster. This data goes to another company’s servers, may be stored there, and sometimes ends up in the training of future models. It can also happen another way: the company’s AI assistant is connected to documents and by mistake shows a person something they should not see. Protection starts with simple rules: what may be sent to a neural network, what may not, and which services to use.

For business

The consequences are measured in money: fines for personal data breaches, loss of trade secrets, and customer claims. The sources of risk are public services that employees use without permission, assistants with too much access to files, and integrations that give the model more fields than it needs. The measures, from cheapest to most costly, are: an AI use policy and employee training, corporate plans that opt out of training on your data, masking personal data before it is sent, and your own model in a closed network for highly sensitive data. Start with an inventory: which services employees already use and what goes into them.

For a specialist

OWASP lists sensitive information disclosure among the key risks of LLM applications, item LLM02 in the 2025 edition: personal, financial, and medical data, business data, credentials, and legal documents can appear in the model’s output. The channels are data sent to an outside API, prompts stored by the vendor, memorization of training data, overly broad assistant access in RAG and agent systems, and prompt injection through connected documents. Carlini et al. (2020) showed that word-for-word fragments of the training set, including names, phone numbers, and email addresses, can be extracted from GPT-2, and that larger models are more vulnerable than smaller ones. OWASP measures: cleaning and masking data before training and sending, the principle of least privilege, access control for sources, differential privacy and federated learning, clear retention rules, and a user’s right to opt out of training on their data. Bans in the system prompt are not reliable on their own.

Example

A manager pasted a CRM export with customer phone numbers into a public chatbot to get a summary. After that, the company set up corporate access to the model with phone number masking and banned uploading exports to personal accounts.

Related terms

Questions and answers

How do employees cause AI data leakage? Usually with no bad intent: they paste contracts, customer tables, or code into a public chatbot. The data ends up with an outside vendor and leaves the company’s control.
How do you protect a company from leaks through AI? Adopt an AI use policy, give employees a corporate service instead of personal accounts, and mask personal data before it is sent. For highly sensitive data, use a model inside your own closed network.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us