RAG chunking: how AI search engines decide what to cite

Google indexes your whole page. An AI answer engine never sees it. What gets embedded, ranked, and handed to the model is a chunk, a passage of a few hundred tokens cut out of your post at a boundary you didn't choose. If that chunk doesn't stand on its own, the rest of the page might as well not exist. This is the mechanical reason behind a pattern that trips up a lot of writers: a post ranks #1 in Google and never shows up in a ChatGPT or Perplexity answer. Position Digital's 2026 research, summarized in Authority Tech's citation gap analysis, found 80% of ChatGPT's most-cited pages don't rank anywhere in Google's top 100, and only about 12% of URLs cited across ChatGPT, Perplexity, and Copilot rank in Google's top 10. The two systems are not grading the same homework.

Our answer engine optimization pillar covers the broader practice of getting cited, and the answer-first content structure post covers the on-page checklist: the direct-answer block, the self-contained section, the byline. This post goes one layer under both. It explains retrieval-augmented generation (RAG) as the actual engineering problem it is for a writer: a token-budget problem, decided by how your document gets chunked before an AI system ever reads a word of it.

Why RAG retrieval is not the same as a Google crawl

A Google crawl reads a page. A RAG pipeline reads a chunk. That single difference explains why the two systems can disagree completely about which page deserves the citation for the same question.

What retrieval-augmented generation actually does with your page

Retrieval-augmented generation is the architecture behind every major AI answer engine: ChatGPT, Perplexity, Claude, and Google's AI Overviews and AI Mode. Instead of relying only on what the model memorized during training, the system fetches relevant text at query time and hands it to the model as context before it writes an answer. Perplexity's pipeline, for example, runs hybrid retrieval, combining BM25 keyword matching with dense embeddings, then filters candidates through a multi-layer reranker before the model ever synthesizes a response. The retrieval step, not the writing step, decides what the model is allowed to say.

Perplexity's own founder describes the grounding rule as stricter than plain RAG. "The principle in Perplexity is you're not supposed to say anything that you don't retrieve," Aravind Srinivas said on the Lex Fridman podcast, "which is even more powerful than RAG because RAG just says, okay, use this additional context and write an answer. But we say, don't use anything more than that too." If your content never gets retrieved, it isn't a candidate for the answer. It's invisible before the model starts generating.

The chunk is the unit of retrieval, not the article

Before any of that retrieval can happen, your document has to be cut into pieces. A vector database can't embed a 3,000-word post as one object and expect a similarity search to work well against a specific question, so every RAG pipeline splits source documents into chunks, usually a few hundred tokens each, and embeds each chunk separately. When a query comes in, the system compares the query's embedding against every chunk's embedding and pulls back the closest handful of matches, not the whole document.

That means the article is not the thing being judged. The chunk is. A post can be long, well-researched, and internally consistent, and still lose the retrieval contest if the one paragraph that answers the query got cut awkwardly, or got buried in a chunk that's mostly about something else. Google's own AI Mode works the same way at the retrieval layer: it runs query fan-out, splitting one question into many sub-queries and retrieving passage-sized chunks for each in parallel, rather than reading one page top to bottom.

Why a #1 Google ranking can still be invisible to ChatGPT or Perplexity

A #1 ranking measures something different: whole-page authority, backlinks, and historical engagement signals aggregated across an entire URL. RAG retrieval measures whether one 300-token slice of that URL is the closest embedding match to one specific question. Those are different tests, run by different systems, and a page can pass one and fail the other completely.

The gap shows up in more than one dataset. A separate 100-page analysis found citation rates climbing sharply with rank, position 1-3 pages got cited by ChatGPT 83.3% of the time versus 44.0% for position 11 and below, so ranking well still helps on average. But "helps on average" and "guarantees a citation" are not the same claim, and the 80%-of-citations-outside-the-top-100 finding above shows just how loose that correlation actually is once you're looking at what a specific answer engine chose to quote.

The three chunking strategies and which one AI systems reward

Whoever built the retrieval pipeline that might cite your post had to choose a chunking strategy, and the choice was not neutral. There are three main approaches, and the one with the best reputation is usually not the one that wins in practice.

Fixed-size chunking: simplest, and a stronger baseline than it gets credit for

Fixed-size chunking splits text at a raw token or character count with no regard for sentence or paragraph boundaries: pick a number, split when you hit it, repeat, usually with some overlap carried into the next chunk to preserve a little context across the cut. It sounds crude, and it is, but it's cheap to run and doesn't require an embedding call just to decide where to cut.

Recursive chunking: splitting at natural boundaries first

Recursive chunking tries to respect the document's own structure. It checks a hierarchy of separators in order, paragraph breaks first, then line breaks, then sentence boundaries, then word boundaries, and cuts at whichever one gets it closest to the target size. Firecrawl's chunking guide recommends it as the default for most RAG use cases, landing around 400-512 tokens with 10-20% overlap. It's still mechanical, just mechanical with better manners.

Semantic chunking: the expensive strategy that usually loses

Semantic chunking is the strategy that sounds the smartest: embed every sentence, compare each one to its neighbor, and cut where the topic shifts rather than where a counter hits a number. It requires an embedding call per sentence during ingestion, which makes it the most computationally expensive of the three, and the evidence says that expense usually doesn't pay for itself.

What chunk size actually wins, with the benchmark numbers

Two independent benchmarks point the same direction. A Vectara study published at NAACL 2025 tested fixed-size against semantic chunking across multiple datasets and found fixed-size matched or beat semantic chunking on natural-language corpora, 90.59% F1 on HotpotQA and 93.58% on MSMARCO, against fixed-size's weaker 69.45% on the Miracl dataset, where breakpoint-based semantic chunking won at 81.89%. The paper's own conclusion: semantic chunking's occasional gains don't justify its added compute cost for most real-world corpora.

A newer, more granular benchmark makes the same case with sharper numbers. Testing 7 chunking strategies across 50 real academic papers totaling roughly 906,000 tokens, recursive character splitting at 512 tokens won outright at 69% end-to-end answer accuracy, ahead of fixed 512 tokens (67%), fixed 1024 tokens (61%), page-per-chunk (57%), semantic chunking (54%), doc-structure-aware chunking (52%), and proposition chunking (51%). Semantic chunking placed fifth of seven, ahead of doc-structure-aware and proposition chunking but well behind both plain recursive and fixed-size splitting, despite being the most expensive strategy to run. The reason it underperformed is concrete and a little absurd: it produced 17,481 chunks averaging just 43 tokens each, fragments too small to carry a complete idea, from the same 50 papers. The strategy built to preserve meaning fragmented it instead.

(Source: runvecta.com's 2026 chunking benchmark, 50 academic papers, ~906,000 tokens.)

The pattern holds beyond this one study. Milvus's guidance on chunk size puts 128-256 tokens as the sweet spot for fact-based, precise queries, and 256-512 tokens for broader questions that need more surrounding context. Either way, the winning range sits well under a full article, which is the whole reason chunk-level writing matters more than total post length.

Why chunk boundaries destroy context, and what fixes it

Getting the chunk size right solves one problem. It doesn't solve the other one: even a well-sized chunk can lose the information that made it meaningful in the first place, because the cut happened somewhere your reader was relying on the sentence before it.

The lost context problem: a chunk that doesn't know what it's about

Anthropic's own research on the failure mode is blunt: "Traditional RAG systems have a significant limitation: they often destroy context," the company wrote in its Contextual Retrieval announcement. Its example is the one every writer should picture happening to their own posts: a chunk stating "the company's revenue grew by 3% over the previous quarter" is useless on its own, since it doesn't say which company or which quarter, details that lived in a sentence one paragraph up that got cut into a different chunk.

The same mechanism explains why long chunks that do survive retrieval can still underperform short, well-targeted ones. Research on the "lost in the middle" effect found that model performance is highest when relevant information sits at the start or end of a retrieved context, and degrades when it's buried in the middle of a long passage, even in models built for long context windows. A short chunk sidesteps the problem by never burying the answer in the middle of anything.

Anthropic's Contextual Retrieval fix, and what it implies for writers

Anthropic's fix for the lost-context problem is to prepend a short, model-generated explanation to each chunk before embedding it, restating what the chunk is about and how it relates to the rest of the document. The results were substantial: contextual embeddings alone cut the top-20 retrieval failure rate from 5.7% to 3.7% (a 35% reduction), adding contextual BM25 brought it to 2.9% (49% reduction), and adding a reranking step on top brought it to 1.9%, a 67% reduction from baseline.

The implication for a writer isn't "go build a reranker." It's that the fix Anthropic had to engineer at the infrastructure level is the same problem you can solve at the writing level, for free, by never depending on a pipeline to reattach context your own sentence should have carried. If Anthropic had to build a system to reintroduce the company name and the quarter, the cheaper fix is writing the section so the name and the quarter were never separable from the number in the first place.

A structural checklist: rewriting posts into retrievable, self-contained passages

None of this requires a rewrite of your entire editorial process. It requires treating each section as the actual unit that gets read, because for a retrieval pipeline, it is.

Size sections to the chunk window, not the word-count goal

Stop budgeting sections in "roughly 300 words" and start budgeting them in tokens, since that's the unit the retrieval system actually counts against. A token is roughly three-quarters of an English word, so a 512-token window is around 380 words, and the benchmarks above show 512 tokens beating both smaller fixed windows and the 1024-token option. Write each section to resolve fully within that range: one claim, its support, and nothing left dangling for the next section to finish.

Resolve every pronoun and cross-reference inside the section

If a sentence in section four only makes sense because you defined a term in section one, a chunking pipeline that cuts between the two has just made section four unusable. Reread each section and replace "it," "this," and "as discussed above" with the actual noun, every time. This is the same discipline our answer-first structure post recommends for the direct-answer block, applied to the whole section instead of just its opening lines.

Put the answer, the entity, and the number in the same 200-400 token span

Anthropic's revenue example works as a warning specifically because the entity (the company name), the number (3%), and the time frame (the quarter) got split across a boundary. Keep them together on purpose. Name the entity again instead of using a pronoun, attach the dated, sourced figure to the same sentence as the claim it supports, and don't let scope creep push a second idea into the same section. Our docs SEO post covers this same principle for API reference pages, where the cost of a broken reference is even higher because there's no narrative to fall back on.

Test it: the delete-everything-else check

Take one section, delete every other section in the post, and read it in isolation. If it still makes complete sense, states its claim, and needs nothing from the rest of the page, it will survive being chunked and retrieved on its own. If it depends on a definition, a number, or a name introduced somewhere else, a retrieval pipeline will eventually cut it exactly where it hurts. Glossary-style entries pass this test almost by construction, which is why they're worth studying even if your post isn't a glossary.

Here's the check run against a real chunk instead of a hypothetical one. Pull this two-sentence passage out of our docs SEO post and delete everything else around it: "Call POST /v1/keys/rotate with your current key in the Authorization header. The response returns a new key and invalidates the old one after a 24-hour grace period, so in-flight requests don't fail mid-rotation." It survives. The endpoint, the header, and the behavior are all named inside those two sentences, so a retrieval system that grabbed only this chunk would have everything it needs to answer "how do I rotate an API key" without borrowing a word from the page around it. Now run the same isolation on Anthropic's own warning example from earlier in this post, "the company's revenue grew by 3% over the previous quarter", and it fails immediately, which is exactly why Anthropic picked it: no entity, no date, nothing a retrieval system could ground an answer in. The difference between those two chunks is the whole checklist above, applied.

Getting this right on one post is a rewrite. Getting it right on every post, section by section, on a schedule, is the actual discipline, and it's the same discipline that makes a machine-readable layer like llms.txt worth adding on top rather than instead of it. The file points a crawler at your docs; it doesn't make a badly chunked page retrievable once it's there.

Lyra writes every section sized and self-contained enough to survive being chunked and retrieved on its own, the exact discipline this post lays out, then opens the post as a pull request you review before it ships.

Try Lyra → · see the plans