TerminalServer RDP session

When I want to make an RDP session from one server 2016 to another server 2016, the first connection takes about 45 sec. If I try again right after the successful connection, it takes only 15 sec. After some time, it takes 45 sec again.

In the EventLog I see that between EventID 1029 Base64(SHA1(UserName) and EventID 1102 The client has initiated a multi-transport connection to the server it takes so long. How can I find out what the server is waiting for. I already analyzed it with Wireshark and even there I don't see anything except that it waits these 45 sec.

Does anyone have any idea?

Thanks a lot