Windows Server 2019 memory utilisation

The memory usage goes up to more than 98% without using significant resources on the Server. When the Server is restarted, the memory usage is normal as indicated in screenshot 1. But after a while without using any resources, the memory usage goes up to 84%. How can we overcome this problem? How can we reduce the memory utilization when the machine is not utilised fully?