"Shared with Us" link in Modern SharePoint Menu (behavior)?

Just responding to your original post of three years as I came across the same issue. Until Microsoft provides a solution to remove Shared with Us link, you might or might not be aware of removing "Quick Launch".  I've worked around the issue by removing Quick Launch altogether via Settings/Site settings/Navigational Elements and unchecking "Quick Launch" check box. The kicker here is, you lose the ability to navigate to various sections from left navigation. It's a price to pay if you don't understand the behavior of Shared with Us and it's  getting on your nerves.

I created temporary navigational links on my Home page using Quick Links, Button and Call to action Web Parts. Unfortunately web parts can't be applied  on document libraries so you are stuck with using the browser back button or site logo to navigate back. Just another option if it is an option. Brent Ellis