To get free/busy information from users calendars (calendar: getSchedule - Microsoft Graph v1.0 | Microsoft Docs), the API permissions Calendars.Read or Calendars.ReadWrite are required. Both allow quite a lot of access to the users calendars.
It would be great to have a more granular API permission just to read free/busy information. This would allow applications to help with meeting scheduling without exposing all calendar details.