@natali320 I know your post is old but maybe this will help you/someone else in the future.
I had the same problem and it isnt exactly random it is usually the first time the query is made as it makes a call to msoid.[yourdomain].co.uk and msoid.co.uk which dont exist so the wait time is due to it waiting for that connection to time out which is usually 20 seconds - a couple of mins before it actually makes the calls to pull the data from sharepoint which that part is actually really quick
You should be able to see this yourself if you run a program like fiddler (https://www.telerik.com/download/fiddler) which allows you to see all the web calls your server is making. Start a trace in fiddler and you'll see right at the point where your code calls the csom ExecuteQuery command it will take forever to realise it cant get to those msoid domains, the 2nd time you make a ExecuteQuery call it will be much faster due to it knowing msoid domains arent accessible
To fix this I just had to add those domains in my hostfile to point to my loopback address so it fails straight away