Data Leakage

Data leakage in AI systems happens at three layers. At training time, models can memorise rare strings from their corpus — phone numbers, passwords, API keys committed to public code — and an attacker who knows the right context can prompt the model to regurgitate them. At inference time, applications often pass sensitive context to third-party APIs (OpenAI, Anthropic, Bedrock) without redaction; this content is then potentially logged, retained, or used to improve future models depending on the vendor's terms. At the application layer, multi-tenant deployments routinely leak across users when caching, logging, or vector-store indexing is misconfigured. Indirect prompt injection compounds all three by giving an attacker a way to ask the model to repeat what it should not. Defenses: PII redaction in prompts and outputs, differential privacy in training, vendor data-use review, and strict tenant boundaries in shared infrastructure.