Model Poisoning

Model poisoning is a training-time attack that leaves the model functionally normal on most inputs but misbehaving on attacker-chosen triggers. The original BadNets paper showed this on image classifiers: stamp a small pixel pattern on a stop-sign image during training, and the deployed model misclassifies any future stop sign with the same pattern as a speed-limit sign. The same idea generalises to LLMs (trigger phrases that flip refusal behaviour), code models (triggers that emit insecure code), and reinforcement-learning agents (reward hacking via tampered reward signals). The attack is hard to detect because standard validation sets show no degradation. Federated learning is particularly exposed because the training data and gradients come from many untrusted clients. Defenses include trigger detection (Neural Cleanse, ABS), spectral signatures, robust aggregation in federated setups, and strict provenance on training data.