Azure 2FA multi factor authentication causing RDS gateway to reached it s maximum allowed connection

After adding 2FA to our RDS gateway windows 2012 R2, users have been complaining they cannot login because RDG server reached its maximum allowed connections. After checking, the limit is 30 but only a few users are in the system. After increasing to 50, a few more users were able to connect. It seems like the connections are not ending properly.