RAG assistant answering from old documents: stale indexes, orphans and drift

Somebody asks the assistant about the returns window and it answers thirty days, confidently, with a citation. The policy changed to fourteen in March. The document was updated, the change was announced, and the assistant has been quoting the old number to customers for five months because nothing connected the edit to the index.

This failure has no error state. Retrieval succeeded, the answer was grounded, the citation is real, and every dashboard is green. Below are the four ways an index falls behind, how to tell which one you have, and a monitor that reports the gap each morning instead of leaving it to a complaint.

1. The document changed and nothing re-indexed it

The most common shape. Indexing ran once at launch, or runs on a schedule that only picks up new files, and edits to existing documents never trigger anything.

source:  returns-policy.md   modified 2026-03-14T09:22Z   hash a91f...
index:   doc_returns_policy  indexed  2025-11-02T14:10Z   hash 4c7e...

assistant: "You can return items within 30 days of delivery." [returns-policy.md]

The citation points at the right document, which is what makes it convincing. What it does not tell the reader is which version of that document produced the sentence. A modification timestamp comparison catches this in one query, and almost nobody runs it, because the pipeline was built to answer "is this document present" rather than "is this document current".

2. The document was withdrawn and stayed in the index

The dangerous one. A policy is deleted or superseded at the source and the vectors remain, so the assistant keeps answering from a document the company has explicitly retracted.

[retrieval] doc_8812 "pricing-2025-promo.md" (0.87)   // deleted at source in January
[assistant] "That plan is $49£40€45 a month with the launch discount applied."

Deletions rarely propagate because most indexing jobs are additive: they walk the source, upsert what they find, and have no step that asks what is in the index but no longer in the source. An orphan is worse than a stale document, since a stale one is at least a real policy that used to be right, while an orphan can be something legal asked to be taken down.

3. Chunks from the old version survive alongside the new

A subtler variant of the first two. Re-indexing ran, but chunk identifiers are derived from position or content rather than from the document, so the new version was added and the old chunks were never removed. Retrieval now sees both, and the two contradict each other.

[retrieval] chunk 4471 "returns within 14 days" (0.81)   from v3
[retrieval] chunk 2209 "returns within 30 days" (0.79)   from v1, orphaned

[assistant] "Returns are accepted within 14 to 30 days depending on the item."

That sentence exists in no document. The model was handed two contradictory passages and did what a helpful reader does, which is reconcile them. The tell is an answer that is oddly hedged or gives a range where your policy gives a number. The fix is deletion by document identifier before upsert, so a re-index replaces rather than accumulates.

4. The document is current and retrieval never reaches it

Worth separating out because the remedy is completely different. The index is fine; the right chunk sits below the cutoff, and an older document that matches the question's wording more closely wins.

query: "how long do I have to send something back"
  0.71  faq-shipping.md        "...send back within 30 days..."   [2024, superseded]
  0.68  returns-policy-v3.md   "...fourteen days from delivery..." [current]
  --- top_k = 1 ---

Six weeks of prompt engineering will not fix this, because the prompt is not the problem. Raising top_k and adding a reranking pass usually is, and so is deleting the superseded FAQ that should not have been competing in the first place. Which brings the diagnosis back to the same question as the other three: does the index contain exactly what the source contains, and nothing else?

Tell them apart before changing anything

The four faults need four different fixes and they present identically, so the ordering matters. Ask, in this sequence: does the index hold a version of this document that matches the source hash; does the index hold documents the source no longer has; does retrieval return more than one version of the same document; and only then, is the right chunk being retrieved at all. The first three are answered by comparing two lists, which takes minutes. The fourth is real retrieval work and is worth doing only once the first three are clean, because chunks from withdrawn documents will otherwise poison every experiment you run.

Check it every morning, not after a complaint

The attached workflow fetches your source listing and your index listing, compares them, and reports the three kinds of drift separately: documents never indexed, documents whose index copy is behind the source by more than a day or whose content hash differs, and documents in the index that no longer exist at source. It alerts only when something has drifted, and records a clean run when nothing has, so the execution history answers "when was the index last verified" without anyone having to remember.

It needs two things shaped by you, both marked in the workflow: a listing of source documents as { id, title, updatedAt, hash }, and the equivalent from your vector store. Pinecone, Qdrant and pgvector can all produce that metadata. The workflow validates clean against the current n8n schema, which we check before publishing anything importable.

When not to hire anyone

If your knowledge base is a handful of documents that change a few times a year, you do not need a monitor. You need a line in whatever process changes a policy that says re-index, and someone to run the comparison by hand each quarter.

If you have found one stale document and can re-index it, do that. One wrong answer traced to one document is not a project.

Where it becomes worth paying for is when documents change weekly, several people can edit them, and nobody can currently answer how many of the assistant's sources are current. At that point the fix is not a re-index, it is making the indexing an automatic consequence of the edit and having something that notices when it is not, which is a change to the pipeline rather than to the data.

What it costs to have us do it

A written diagnostic is $500£400€450, delivered in two business days, credited in full against whatever follows and refunded if it names no fixable cause. Here it means running the comparison against your live index and coming back with a count: how many documents are current, how many are behind, how many are answerable but withdrawn, and which of the four faults produced the answer that sent you looking.

Repairs land at $750£600€700 to $1,500£1,200€1,400 where the indexing job needs deletion-before-upsert and a freshness check, and $1,500£1,200€1,400 to $4,000£3,100€3,700 where the pipeline needs rebuilding so an edit at source triggers indexing on its own. Both leave the monitor running, because an index that is correct today drifts the first week nobody is watching.