Why Copilot Gives Wrong Answers: Stale SharePoint Content, Not Bad AI
A sales lead asks Microsoft 365 Copilot who the new VP of Customer Success is. Copilot replies with the name of the person who left four years ago, and cites a 2019 org chart. Two weeks later it routes a procurement request to an approver who left 18 months earlier. Then it tells a new starter that meal expenses are capped at $25, even though HR raised the limit to $40 years ago.
This is a real (anonymised) rollout I helped review. The CIO’s first question was: “Is Copilot broken?” It wasn’t. Copilot did exactly what it is designed to do. It retrieved the most relevant content the user had access to and answered fluently. The problem was the content.
In this post I explain why Copilot gives wrong answers when your SharePoint is full of stale material, the patterns I see again and again, and the practical clean-up plan I now recommend before, or straight after, any Copilot rollout.
What the investigation actually found
When we traced each wrong answer back to its source, there wasn’t a single Copilot bug. There was a tenant that had been allowed to drift for six years:
- The 2019 org chart was still in the HR site, in a folder called “Historical”. Folder names mean nothing to Copilot. If a file is indexed and the user can open it, it can be used as grounding.
- The 2020 expense policy sat in the policies hub next to the current version. Both were searchable, and the old one had the more “official” file name.
- The intranet Leadership page used an old org chart image. The image had alt text, and Copilot read the alt text.
- A former approver was still named as the approval contact in 14 procurement documents that had been versioned for years without anyone updating that field.
None of these problems was new. For years, people simply asked a colleague and got the right answer. The wrong answer was sitting in SharePoint the whole time, but nobody read it. Copilot reads everything.
Wrong answers are a data problem, not a hallucination
When people say Copilot “hallucinated”, they usually picture the model inventing facts. In most tenants I review, that is not what happens. Copilot grounds its answers in your documents, emails, chats and meetings. It has no built-in way to know whether a file is current, superseded or forgotten. If the file exists and the user can access it, it is fair game.
That is the classic ROT problem: content that is redundant, obsolete or trivial. Before Copilot, ROT was mostly a storage cost and the occasional wrong file opened by accident. With Copilot, ROT becomes something the AI actively retrieves and presents with the same confidence as your current, correct content.
Three patterns that cause most wrong answers
1. Stale policies
Two versions of the same policy live side by side, for example an old expense policy and the current one. Nothing marks the old one as superseded, so Copilot may quote whichever ranks higher for the question.
2. Duplicates that drift apart
Sales and Marketing each keep a copy of the price sheet. Sales updates theirs to v1.1 and Marketing doesn’t. A Marketing account manager asks Copilot for current pricing and gets v1.0. The deal closes at the wrong number.
3. Orphaned content
The wiki from a 2018 acquisition is still in the tenant, and nobody owns it. A support agent asks about a feature, Copilot finds the answer in that wiki, and the customer is told about a feature that was retired years ago.
In every case the language model is doing its job. The data is doing the lying.
Why Copilot makes stale content more dangerous
Before Copilot, stale content was passively bad. It only caused harm if someone stumbled across it. Copilot changes how content gets consumed: every question pulls in candidate content, ranks it and serves it. The pile of “files nobody looks at anymore” effectively disappears, because Copilot looks at all of them.
There is also a slower, more damaging cost: trust. After a few incidents like the ones above, users start double-checking every answer, and adoption stalls. The tool gets labelled “unreliable” when the real issue is the data underneath it. I cover the human side of this in Copilot readiness is human, not technical.
Security readiness vs data-quality readiness
Most Copilot readiness work focuses on oversharing: who can see what they shouldn’t. That matters, but it is only half the job. A tenant can have tight permissions, good DLP and sensitivity labels, and still produce unreliable answers.
| Security readiness | Data-quality readiness | |
|---|---|---|
| Question it answers | Who can access what? | Is what’s accessible still true? |
| Typical risk | Oversharing and data exposure | Confident answers from outdated content |
| Typical fixes | Permissions clean-up, sensitivity labels, DLP | Archive, delete, deduplicate and assign owners |
| Who notices first | Security and compliance | End users, then the CIO |
Copilot needs both questions answered before you can trust it. For the technical foundations, see Before you turn on Copilot: the technical foundations you cannot skip.
A practical clean-up plan
Microsoft’s long-term answer is information protection and lifecycle management, which is correct. But that is a multi-year programme. In the meantime, this is the faster approach I recommend:
- Inventory the tenant. Scan SharePoint, OneDrive and Teams for content that hasn’t been modified or opened in years, near-duplicate documents and sites with no active owner.
- Prioritise what Copilot is most likely to cite. Policies, procedures, price lists, org charts, templates and “how we do X” pages matter far more than old project files.
- Archive or delete obsolete material. Use retention policies and archiving rather than leaving superseded versions live and searchable.
- Make the authoritative version obvious. Keep one current copy, link to it from everywhere else, and remove the duplicates.
- Give every site and library an owner. Orphaned content is where wrong answers hide.
- Temporarily exclude risky sites from Copilot discovery while you clean them, rather than delaying the whole rollout.
- Re-scan regularly. Content drifts back. Make the scan a quarterly habit, not a one-off project.
Here is what a content scan looks like in practice. These screenshots come from a demo tenant scanned with LogiSam’s Tenant Storage Optimisation. You can produce the same inventory with any reliable reporting tool or PowerShell.
What happened after the clean-up
In the story above, the first scan produced 1,847 recommendations. The team didn’t try to fix everything. They handled the top 200, which covered the content types most likely to be cited by Copilot. Three months later there had been no new Copilot incidents, and their storage bill had dropped by 19% as a side effect.
The lesson I share with every client: the clean-up has to happen either way. Do it before you enable Copilot and it is a calm, planned project. Do it afterwards and it happens in incident reviews.
FAQ
Does Copilot know which version of a document is current?
Not reliably. It ranks content by relevance and other signals, but it has no built-in concept of “superseded”. If old and new versions are both accessible, either can be used as grounding.
Do sensitivity labels stop Copilot from using outdated content?
No. Labels control protection and access, not accuracy. An outdated policy with a “General” label is still an outdated policy.
Should we delete or archive stale content?
Delete true duplicates and trivial files. Archive anything you must keep for legal or records reasons, so it is retained but no longer part of everyday search and Copilot grounding.
How long does a clean-up take?
The scan itself usually takes under an hour. A focused clean-up of the highest-risk content can be done over a few weeks and staged over a quarter.
Related reading: Why most Copilot projects fail · Copilot licensing explained · Stop using folders in SharePoint
