Significant delay between AAD Roles permissions and MSGraph access.

We are using AAD Roles (or even feature level permissions) to give certain Service Principals permissions as they offer less permissions than the relevant MSGraph scopes as we would like to use least privilege principles. However I noticed there is a significant delay between applying an AAD role to a user and that user being able to query the MSGraph API successfully afterwards. We have a case where even after 12 hours after assigning the relevant AAD roles, MSGraph is still throwing unAuthorized errors on requests. 

Is this due to some cached token lifetime somewhere where MSGraph does not look at this user's AAD roles for a while until this token expires or something?