Users can access Planner even if their license has been disabled?

We noticed a few weeks ago, and confirmed via a Microsoft Support Ticket, that our users can still access Planner even after we've disabled their license for Planner.  The license switch apparently o...