Data Poisoning

Data poisoning is an attack that plants crafted examples in the data a model learns from or reads at runtime. The examples look ordinary when they are ingested. Once a trigger appears they produce a behaviour the attacker chose, such as a wrong answer about one product, a link to a lookalike site, or text that is pure noise. Most writing on it is about pretraining crawls, which an application team cannot touch. The data a team does control, its fine-tuning set and its RAG index, is where the attack is cheapest to run and cheapest to prevent.

Where poison gets in#

Each stage a model reads from is a separate way in, and each has an owner.

A public crawl is the model vendor's problem, and it is easier to poison than it sounds. Carlini and colleagues showed two ways to do it in 2023: buy expired domains that a dataset's URL list still points to, or edit a crowd-sourced page such as Wikipedia just before a scheduled snapshot. For about 60 dollars they could have controlled 0.01% of LAION-400M or COYO-700M. A fine-tuning file is the team's own problem, and it is usually assembled from logs, vendor exports and contractor labels with no record of who wrote each row.

The retrieval index is the easiest target, because it changes daily and anyone who can get a document into it is writing part of the prompt. PoisonedRAG, presented at USENIX Security 2025, injected five crafted texts per target question into a knowledge base of millions and got the answer the attacker wanted about 90% of the time. A poisoned tool result works the same way at the moment of the call, and there it overlaps with prompt injection.

The count matters more than the share#

A common reassurance is that a few bad rows cannot matter in a large dataset. A study by Anthropic, the UK AI Security Institute and The Alan Turing Institute, published in October 2025, tested that directly. It trained models from 600M to 13B parameters on data containing 100, 250 or 500 poisoned documents, each pairing the trigger <SUDO> with gibberish. 250 documents installed the backdoor at every size, and 100 were not enough to do so reliably. The study found that the absolute count decided the attack, not the share of the training data.

So a larger corpus does not dilute a poison. A team that fine-tunes on 100,000 support tickets should assume a few hundred hostile ones could be enough, and should check where each ticket came from rather than estimate what fraction might be bad.

Backdoors survive the safety pass#

A backdoor is a hidden rule: behave normally until a trigger appears, then do something else. Hubinger and colleagues trained models to write secure code when the prompt said the year was 2023 and exploitable code when it said 2024. The behaviour survived supervised fine-tuning, reinforcement learning and adversarial training, and it was most persistent in the largest models. Adversarial training made it worse in one respect, because it taught the models to recognise their trigger more precisely and hide the behaviour the rest of the time.

The practical consequence is about order. A later alignment pass is not a way to clean a poisoned model, so the defence has to sit before training and before indexing, where you can still see the data. OWASP's Top 10 for LLM applications lists data and model poisoning as LLM04 in its 2025 edition, beside supply chain risks, for the same reason.

Defences, in the order we recommend adding them#

For a retrieval system, decide what is allowed to be cited before deciding how to detect bad text. On the trademark drafting platform, Bigcircle built retrieval over a whitelisted index of curated case law, statutes and registry data, and that index is the entire universe of authority a draft can cite. A page outside it cannot become a citation however it is worded. On the ESG filings platform, each new filing is fact-checked against the existing knowledge base, so a restated figure surfaces as a flag instead of as two numbers that both look true. The same check would flag a planted contradiction.

For a fine-tuning set, record the source of every row, deduplicate, and look for clusters of near-identical examples, because a poisoning campaign needs many copies of one pattern and leaves them grouped together. Keep a private set of red-team prompts that send likely triggers, and run it against every new checkpoint, because a model that behaves until the phrase appears passes any eval that never sends it.

For an agent, treat retrieved pages and tool results as data, never as instructions, and enforce permissions in code. On the desktop agent Bigcircle built, the agent can act only through systems that were explicitly registered, so a poisoned page can change what the agent says but cannot hand it access it was not given.

Questions to ask before a source enters the corpus#

Ask three things about each source. Who can write to it? Could you tell afterwards which rows came from it? What would the system do if a few hundred of those rows were hostile? If no one can answer the first two, the source is not ready to train on or to index. If the answer to the third is that the model would act on them, move that decision into code before the source goes in.