Checked on OWA and there isn't any performance issue
Recreated Outlook profile on Mac Outlook 2016 but still issue remained same
Checked for Outlook update and it said it's latest version
Cleaned Cache for Calendar and it made little bit better performance from 4 minutes to 40-50 seconds while clicking on Mail Tab to Cal Tab
However the issue is still slower performance compare to OWA and other Outlook On Ipad/Iphone devices.