AI Package Hallucination Persists; Perplexity Ships Free MCP Scanner
Frontier LLMs still hallucinate package names at 4.6–6.1% across the current cohort — compressed versus 2024 findings but not resolved. The structural risk is sharper: a new study finds 127 package names invented identically by all five evaluated models, creating a model-agnostic slopsquatting surface no single-vendor audit can reveal. Perplexity responded this week by open-sourcing Bumblebee, a read-only security scanner for developer environments that specifically targets MCP configuration files.
What the Source Actually Says
A replication study by independent researcher Aleksandr Churilov (arXiv 2605.17062) ran 199,845 paired Python and JavaScript prompts across Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2. Hallucination rates fell sharply from Spracklen et al.'s 2024 benchmark (5.2–21.7%), but not to zero: 4.62% for Claude Haiku 4.5 to 6.10% for GPT-5.4-mini. The critical finding is the 127 cross-model identical hallucinations — 109 on PyPI, 18 on npm — that all five models invent regardless of vendor. A single-model audit clears your vendor stack without detecting this shared blind spot, leaving a pre-seeded slopsquatting attack surface that no model swap can remove.
Concurrently, Perplexity open-sourced Bumblebee — the tool it uses internally to protect its own team. Bumblebee scans browser extensions (Chrome, Edge, Brave, Arc, Firefox), VS Code-family editor plugins, packages across npm, PyPI, and Go, and MCP config files — described as "the local settings that tell AI assistants which external services they can access, including emails, databases, and code repos." The scan is read-only and executes nothing, preventing it from triggering the malicious code it seeks. The release responds to a documented attack where a single group poisoned over 160 packages, including a React library with roughly 12 million weekly downloads.
Strategic Take
Any team running MCP-connected agents has compounded blast radius from supply-chain attacks: a poisoned MCP config or hallucinated package can expose the full agent stack. Run Bumblebee (free, Apache 2.0) across developer machines. The 127 cross-model identical hallucinations are a named threat class — add package-lock verification to any CI pipeline that consumes LLM-generated code.
