Copilot for Microsoft 365 Tech Accelerator
Feb 28 2024 07:00 AM - Feb 29 2024 10:30 AM (PST)
Microsoft Tech Community

Concerning Consistent Hallucinations

Iron Contributor

Every once in a while when I try to have BCE provide a page summary for a PDF using the Edge Bar, it'll come up with information about a Korean Room-temperature superconductor (LK-99). I have never searched about this and I'm not sure why it is coming up. At first I thought it was a fluke and that it misinterpreted some information from the document but I've had at least 3 occurrences now..

agreca_0-1699902711808.png

Another instance below shows that it was partially correct when returning info from the page but then in the same response, it lists the URL to the phys.org page talking about the superconductor. I was actually able to reproduce the issue twice with this same document after closing Edge completely. (identifying info has been redacted)

 
 

agreca_5-1699904000250.png

Edit: I should also mention that the document doesn't contain anything pertaining to this superconductor or any references to the article mentioned.

5 Replies

@agreca Experiencing the exact same issue. 

MikevZ_0-1701247175185.png

 

Thank you for bringing this to our attention. We are currently investigating this.
Hi @agreca and @MikevZ because commercial data protection is applied and chat data isn't stored, we're unable to see exactly what might be triggering Copilot to hallucinate in this way. If you're using Creative mode in Copilot, hallucinations are more likely. We recommend that you switch to Balanced mode in Copilot when asking for a page summarization to get a more accurate response.
Hi @Eric_VanAelstyn, I almost always use Precise and this was the case when I received these results.

Hi @Eric_VanAelstyn , I am able to almost consistently reproduce this issue with a Statement of Work document that has no mention of this LK-99 superconductor. What's strange is it happens when Copilot is trying to build a reference to info it finds in the document. See below..

agreca_0-1701441058380.png