There's absolutely something strange going on with this function. I configured this for my dev tenant this week and all looked fine at first for users in both the Outlook Windows app (Version 2105 [14026.20246]) and Web app. In the Windows app I could see a notification when creating a appointment or meeting that reads something like "Your organization is shortening events per default. Show settings [link]" (translated from Swedish).
I did go ahead and clicked on "Show settings", which brought up the standard Calendar settings where everything looked fine as well. The checkbox for "Shorten appointments and meetings" were checked and configured to "End early".
But then, after just browsing the settings and eventually clicking "Cancel", new meetings isn't shortened per default anymore and the notification message has disappeared. Going back to the Calendar settings I see that the checkbox for "Shorten appointments and meetings" is magically unselected and won't enable unless I manually select it. I also restarted the computer and reinstalled Office completely but without any success. The setting remains off.
Anyone else experiencing this?