Group By on Modified By Issue

We have a team of users who are using "group by" on the "modified by" field in a SharePoint Online document library.  The users are regularly reporting issues with this as the group by is mixing users together.

• SharePoint Online document library
• Library as a whole exceeds the 5K view limit but the group by is in a folder with a handful of files

I can't post a screen grab but the group by looks like this:

Name,  Modified,  Modified By

#Modified By:  User 1 (2)
    File 1, Yesterday at 20:22,  User 1
    File 2, Yesterday at 20:22,  User 1
#Modified By:  User 2 (5)
    File 3, Yesterday at 20:22,  User 2
    File 4, Yesterday at 20:22,  User 2
    File 5, Yesterday at 20:22,  User 2
    File 6, Yesterday at 20:22,  **User 3**
    File 7, Yesterday at 20:22,  **User 3**
#Modified By:  User 4 (2)
    File 8, Yesterday at 20:22,  User 4
    File 9, Yesterday at 20:22,  User 4

Any ideas what could be causing issues here?