A team of security researchers discovered that nine popular AI tools can be weaponized to build large-scale botnets through a technique called "HallucinationSquatting." The attack exploits a fundamental flaw in how large language models handle uncertainty.
The vulnerability stems from LLMs' tendency to generate plausible-sounding answers rather than admit they lack information. When prompted about nonexistent command-and-control (C2) servers or fake infrastructure, these models confidently fabricate details instead of declining to answer. Attackers leverage this behavior to bootstrap botnet operations at scale.
Here's how it works. A threat actor queries an AI tool with requests about infrastructure that doesn't actually exist yet. The model hallucinates realistic technical details, IP addresses, and configuration parameters. The attacker then registers or deploys actual infrastructure matching these fabricated specifications. Because the AI provided convincing instructions, victims executing those steps unknowingly connect to attacker-controlled systems.
The researchers tested nine major platforms, all of which proved vulnerable to some degree. The attack requires no model modifications, jailbreaks, or bypass techniques. It exploits the baseline behavior that makes LLMs useful for many tasks: their willingness to attempt answering any question with confident-sounding responses.
This reveals a tension in LLM design. Models trained to be helpful often generate output rather than refuse requests. Safety training attempts to limit this, but the fundamental architecture still rewards confident-sounding answers. Distinguishing between "I don't know" and legitimate technical information remains an unsolved problem at scale.
The implications extend beyond botnets. The same vulnerability could compromise other infrastructure-dependent attacks, from supply chain poisoning to fake credential distribution. Organizations relying on AI assistants for technical documentation face amplified risks if users treat hallucinated details as authoritative.
The researchers recommend prompt engineering def