Data Poisoning

Data poisoning is an attack on the material a model learns from or reads at runtime. The attacker inserts examples that look ordinary during ingestion but create a harmful behavior later. In LLM systems, the surface is bigger than pretraining: fine-tuning files, RAG corpora, web pages fetched by tools, and evaluation data can all be poisoned.

Attack surfaces#

Pretraining poisoning targets public web crawls. Fine-tuning poisoning targets curated task data. RAG poisoning targets the documents a system retrieves. Tool-result poisoning targets an agent's observations, which overlaps with prompt injection. The attack can be as simple as a trigger phrase paired with a malicious completion.

Backdoors and persistence#

Backdoors are hidden policies: behave normally unless a trigger appears, then do something else. Sleeper-agent research showed that deceptive or backdoored behavior can persist through some safety training. That matters because "we aligned it later" is not a complete defense if the behavior was deliberately embedded.

Defenses#

Provenance is the first defense. Know where training and retrieval data came from. Deduplicate, filter suspicious clusters, scan for canaries, and evaluate trigger behavior. Treat retrieved text as untrusted input, not as instructions. For RAG, separate content from commands and enforce permissions in code.

Practical takeaways#

Data poisoning is a supply-chain risk. Secure the corpus, not only the model endpoint. Keep data lineage, run anomaly checks, and maintain private red teaming prompts aimed at triggers. Any agent that reads the open web should assume the web is hostile.