OneDrive Discover

As well as the information provided already, I’d consider tackling this from a user education point of view and teach that this is expected behaviour but make sure not to share documents inappropriately. https://docs.microsoft.com/en-us/sharepoint/delve-for-office-365-admins is not much of an option as it has a big potential impact on related features. There is a way to hide documents from Delve, not sure if its applicable in this case - https://docs.microsoft.com/en-gb/sharepoint/manage-search-schema#BKMK_HideFromDelveSteps.

Also OneDrive is embracing more of this type of functionality, for example with a new recommended view that will highlight documents from people around you, so it’s worth getting used to these features and how they work.