We have been using Co-Pilot for approx 2 months within our company and have had mixed results with using the summarization, searching & synthesis feature of both Word & Teams.
One of our core issues currently seems to stem from the fact that a document or chat transcript becomes too long to correctly summarize, search or use synthesis. In word MS claims CP max's out between 18,000-20,000 words however we faced issues with file sizes as small as 5000 Words. We could overcome the issue by splitting the document in half and resaving. This would enable correct searching & synthesis, but without the users knowing the "actual" limits it's just a frustrating user experience, see below:
We worked recently on a short complementary update to be presented to an IC soon. This time Copilot did not struggle to find the file, but when I asked for a summary it misses one of the most important point mentioned in the document. More worryingly, it also misses it even after requesting for it to look specifically for this element. Thus I am back to the same problem; for the same reason it does not seem we can (yet) trust it to find all the relevant information when we ask it to e.g. look for all e-mails or files about a topic and summarize, it does not look like we can trust it to summarize a document and have reasonable trust that all key points will be spotted - CoPilot in Word App was used for this.
Is this a sizing issue or maybe something to do with indexing whereby splitting the doc and resaving somehow triggers the search syntax to correctly render both documents?
We have had a similar scenario across the board for meeting transcripts - it's seems great for high level summaries but if you want it to go into details it really struggles. On the size issue we have had one hour meetings transcripts were CP is unable to find content. However, splitting the transcript up into multiple pieces provides better analysis but still struggles with the details.
Does anyone else face similar issues?
Can we expect this to improve from MS in the near term?
Are their concrete sizing limitations that I can accurately portray to the users that they can get consistent results?
Happy to provide further information if required and any help on these issues would be greatly appreciated.
