I also don't love this new behavior. I understand that MS wants to consolidate developer efforts, but driving people away from SharePoint proper (where the lists can be seen and worked with in context) is a mistake.
The most frustrating aspects of the change are:
- There's no tenant-wide control; there have been tenant-level opt-outs for less significant changes to the UI, I really don't understand why there wasn't for this.
- The behavior is inconsistent. This is not unusual - the incremental rollout of features so they are intermittently there and not depending on which browser, which account, phase of the moon, etc. In this instance, though, it's been just awful. When a user complains that something is happening and I can't reliably reproduce it, that's a problem.
- The controls to change the behavior are inconsistently available. Across three tenants, I have yet to find a single site where the -ListsShowHeaderAndNavigation property exists. In most team sites, the "Navigation Elements" page exists, and in most of those cases, checking the appropriate box will keep the lists opening in-site. The "Navigation Elements" item doesn't even appear in the Site settings for a communication site, but a quick URL hack (just go to the comm site and add "_layouts/15/navoptions.aspx" to the address) will get you there and the checkbox sometimes does what it's supposed to (personal observation puts effectiveness at about 50%).
At a bare minimum, Microsoft should add something to call out the "open in site" icon so at least users will get some kind of explanation and a way back.