AI generated code accuracy starts before you hit publish: catch hallucinated APIs, phantom packages, and broken imports by running every code sample in CI.
A wrong statistic in a blog post misleads a reader who has to go looking for the correction. A wrong code sample fails in front of them immediately, because the one thing a code block asks a reader to do is copy it and run it. That gap is why AI generated code accuracy needs its own verification step, not just the general claim-checking discipline applied to prose. A hallucinated stat is a bad sentence. A hallucinated import is a stack trace with the reader's name on it, and if the hallucination is a package name instead of a function call, it can turn into a supply chain problem before they've finished reading the post.
This post covers why AI-written code samples fail differently than AI-written prose, what the research says about how often that happens, and how to build a verification gate that catches it before a post ships, the same way Lyra treats every claim and link in a draft as unverified until it's checked.
A hallucinated fact and a hallucinated code sample are the same underlying failure, a model generating plausible-looking output with no step that confirms it against reality, but they cost the reader differently. One is a slow-motion error a careful reader might catch. The other fails on contact.
Reading a blog post is a passive act until a code block shows up. At that point the reader's next move is almost always to copy the snippet into their own editor or terminal, and the code either runs or it doesn't. There's no interpretation gap the way there is with a fabricated statistic, where a reader has to independently decide to go verify a number before they discover it's wrong. A broken import or an API call to a method that was renamed two versions ago throws an error the instant someone tries it, with the post's own name attached to the failure.
That immediacy is also what separates this problem from the tutorial code decay covered in our content pruning guide. Content decay is a snippet that was correct on publish day and went stale later, as a dependency shipped a breaking change on its own schedule. What this post is about is different and earlier in the pipeline: a snippet that was never correct in the first place, because the model that wrote it invented a function, a parameter, or a package that doesn't exist and never did. Decay happens to code after it ships. Hallucination happens to code before it ships, and a pre-publish verification gate is the only thing that catches it in time.
A hallucinated function call fails loudly and the damage stops there: the reader gets an error, not a payload. A hallucinated package name is worse, because installing a package executes code, and an attacker doesn't have to guess which fake name to register. The model already invented one for them.
This is the pattern security researchers call slopsquatting: registering a package name that a large language model is likely to hallucinate in its output, so the next developer who copies the model's install command pulls down attacker-controlled code instead of getting a clean 404. Lasso Security's Bar Lanyado tested the idea directly. Multiple AI models kept recommending huggingface-cli as an install command, which isn't the real Hugging Face CLI package name, so Lanyado uploaded an empty, harmless package under that exact name to see what would happen. It picked up more than 30,000 downloads in three months, and the instructions for installing it turned up in the README of a public Alibaba research repository, meaning a real engineering team at a real company had copied a hallucinated install command straight into a document other developers would go on to copy from too.
This isn't a rare edge case. The research on how often code-generating models hallucinate is specific, recent, and larger in scope than most SEO or content-accuracy studies ever get to be, because code either compiles or it doesn't, which makes it unusually easy to measure at scale.
A 2024 academic study generated 576,000 code samples across two programming languages from 16 popular code-generating LLMs and checked every package name they recommended against the real package registries. Just over 19.7% of recommended packages didn't exist at all, and the study catalogued 205,474 unique hallucinated package names across the full corpus. The rate wasn't uniform across model types: commercial models hallucinated packages at an average rate of at least 5.2%, while open-source models hallucinated at 21.7%, over four times as often. Either number means a blog post publishing more than a handful of install commands without checking each one is publishing at least one phantom package on a long enough timeline.
The 2024 finding was about volume. A 2026 re-evaluation of five current frontier models, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2, found something closer to a pattern than a coincidence: 127 hallucinated package names were invented identically across all five models. As of April 2026, 53 of those 127 names, 41 on PyPI and 12 on npm, were still unclaimed and available for anyone to register. Five different labs, five different training runs, and the same fake names came out the other side, which means an attacker doesn't need to profile any single model's failure modes. A name that one frontier model hallucinates is a reasonable bet that several others will hallucinate too. The paper is a preprint, not yet peer-reviewed, but its numbers line up with independent coverage of the same findings.
Adoption and trust are moving in opposite directions. Stack Overflow's 2025 Developer Survey found that developer trust in AI accuracy fell from 40% to 29% year over year, even as AI tool usage kept climbing. The number-one frustration developers named, cited by 45% of respondents, was "AI solutions that are almost right, but not quite," and 66% said they're now spending more time fixing that almost-right code than they used to. That's the exact failure shape a phantom import or a hallucinated package produces: not obviously broken on first read, wrong enough to cost real debugging time once someone tries to use it.
There's a second-order effect showing up in how developers write code around AI output, too. GitClear's analysis of 211 million changed lines of code found that copy-pasted ("cloned") code rose from 8.3% of changed lines in 2021 to 12.3% in 2024, while the refactoring share of changed lines fell from 25% in 2021 to under 10% in 2024. More code gets duplicated in, less gets cleaned up afterward, which is exactly the pattern you'd expect if more of what lands in a codebase came from a copy-paste out of a chat window or a blog post rather than being written and then revised in place.
The fix isn't a more careful read-through. Eyeballing a snippet checks whether it looks plausible, and plausible is exactly what a hallucinated import looks like, that's the whole problem. The fix is a gate that actually executes the code, the same discipline a docs-as-code workflow already applies to reference documentation before it merges.
Rust's own documentation toolchain treats this as a solved, default behavior rather than an optional check: rustdoc extracts every code block from a crate's documentation and cargo test compiles and runs each one as a real test, so a broken example in a doc comment fails the build the same way a broken unit test would. Nobody reads Rust's documentation examples and judges whether they look right; the toolchain runs them and tells you.
The same pattern exists for Markdown specifically, which is what a blog post actually is under the frontmatter. markdown-doctest extracts and runs JavaScript code blocks straight out of a Markdown file, phmdoctest does the same for Python by generating pytest cases from a Markdown doc, and language-agnostic tools like txm run the check regardless of which language a given snippet is in. None of these require inventing new tooling. They require deciding that a code block in a blog post is an artifact that gets executed before merge, not prose that gets proofread.
A snippet that runs today against whatever version happens to be installed locally isn't the same claim as a snippet that will still run for a reader six months from now. Pin the exact version in the install command, and treat "does this package name and version actually resolve on PyPI or npm" as a separate, mechanical check from "does the code run." The research above is specific about where that check catches real failures: a hallucinated package name will fail a registry lookup instantly, before anyone has to run a single line of the code that imports it.
A useful CI gate for code samples checks three things, and any one of them failing should block the merge, not just log a warning:
| Check | What it catches | How to run it |
|---|---|---|
| Execution | Hallucinated functions, wrong parameters, broken syntax | Run the snippet in a real interpreter or sandbox, not a linter |
| Package resolution | Hallucinated or slopsquatted package names | Query the real registry (PyPI, npm) for the exact pinned name and version |
| Import consistency | Renamed or removed APIs between the snippet's pinned version and current | Diff the snippet's imports against the installed package's actual exports |
This is the same "independent checker, re-run after every fix" architecture that a broader editorial review process for AI content's E-E-A-T already argues for on the prose side: a code-verification pass has to run with no shared context from the drafting step, and it has to re-run after every edit, because a fix to one line can silently break an import two lines down that nobody re-checked.
Verification sometimes comes back inconclusive, not just pass or fail: a package needs a paid API key you don't have, or an SDK call depends on live infrastructure a CI job can't reach. That's a real category, and it needs a real answer instead of a shrug.
If the code sample is decorative, illustrating a concept the prose already explains clearly without it, the safest move when it can't be verified is to cut it. A missing code block costs you nothing but a little visual variety. A code block that looks tested but wasn't costs a reader a debugging session and costs you the trust the rest of the post is trying to build.
If the snippet is load-bearing, the post genuinely needs to show the API call, but it can't run in an automated check, say so specifically: name the exact reason (requires a paid key, depends on a live third-party service, needs local hardware) rather than a vague "results may vary." A generic hedge doesn't help the reader and doesn't force you to actually think through what could be wrong. A specific caveat does both, and it's the same rule fact-checking AI content already applies to any claim that can't be confirmed: unverifiable is a blocker to solve or a limitation to name, never a detail to quietly skip.
Lyra treats every code block the same way she treats every statistic and every link: as an unverified claim until something outside the model has confirmed it. She fact-checks the prose claims in a draft against current sources before it ships, and the same standard applies to a code sample, because a hallucinated pip install line is just a different shape of the same failure a hallucinated statistic already is. She opens a pull request and tags you to review, nothing auto-publishes, so a snippet that can't be verified is a decision you make with the specific reason in front of you, not a guess that shipped quietly.
A hallucinated package name in a blog post is a security incident waiting on a copy-paste. Lyra fact-checks every claim, link, and code sample in a draft before it ever reaches your review.
FAQ
A wrong stat sits inert on the page until a reader happens to check it. A wrong code sample fails the moment the reader does the one thing a code block invites them to do: copy it and run it. There's no reading comprehension gap between the error and the consequence. And if the error is a hallucinated package name rather than a bad function call, the failure can escalate from 'this doesn't work' to 'this installed something I didn't intend,' since an attacker can register that exact name in advance.
A 2024 study that generated 576,000 code samples across 16 popular code-generating LLMs found an average hallucinated-package rate of at least 5.2% for commercial models and 21.7% for open-source models, with 205,474 unique hallucinated package names across the corpus. A 2026 re-evaluation of five frontier models found 127 hallucinated package names invented identically across all five, and as of April 2026, 53 of those names were still unclaimed and available for anyone to register.
Slopsquatting is registering a package name that AI models are likely to hallucinate, so the next developer who copies the model's install command pulls down attacker-controlled code instead of a 404. It works because the attacker doesn't have to guess a name, the model already generated one for them, often the same one across several different models.
Extract every code block from the draft and actually execute it, in a real interpreter or a CI job, rather than reading it and judging whether it looks correct. Pin the exact package version the snippet was tested against and confirm that version resolves on the real registry, not just that the package name exists. If a snippet can't be verified this way, cut it or replace it with one you can run, rather than publishing it with a hedge.
Built by the tool you're reading about
Lyra finds the topics worth ranking for, writes them in your repo's voice, fact-checks every claim, and opens a pull request scored and ready to merge. You review and hit merge. Want to see what she'd write for you? Start free with three posts, no card.
Keep reading
AI content governance means proving who wrote a post, who reviewed it, what changed, and when it shipped, an audit trail a Git pull request already gives you.
AI blogs sound generic without a real guide. What belongs in a brand voice AI content style guide, and how to keep it current as your product evolves.
GitHub App permissions decide what an AI writer can touch in your repo: which scopes to grant, which to refuse, and how to audit or revoke access.