Azure Virtual Desktop - Black Screens on logins - What we've tried so far

Yep certainly ack! Trust me that's not through choice I'm considering doing that, it unsurprisingly is not something we have ever tested, but equally having ended up with most of the other available hosts in drain mode earlier its a case of which is the lesser of the possible evils... scale out the seemingly "reliable" machines in case we need it in anger and these issue persist again tomorrow, or end up with hosts at double their designed user density and being completely CPU and memory bound which is where we ended up today or be stuck 75 to 80% of users hitting black screens and having to resort to Outlook and Teams in a browser (and I don't fancy having to explain to anyone else how OWA works tomorrow... I did that enough times today🤦‍:male_sign:). Even if we only get like double the performance with triple the CPU and RAM capacity then that that has to be better than the same performance but with double the users competing for it.

If we believe them one user reported a 4 hours login delay stuck on a black screen... although to be fair I am a little dubious as they had tried to apparently login, got a black screen, left the machine sat in the session to go on a 3 hour seminar in a meeting room followed by lunch and reckoned it was still stuck as they left it - and to be fair it may have been as the fix was Ctrl-Alt-End and telling it to log off.

Scaling wise we found the opposite; we were using smaller VMs, some of the customer LOB apps were bursty and CPU intensive at time and RAM heavy (generating final documents - oh and opening certain national newspaper's website in breaks that I wasn't convinced weren't using javascript crypto-mining scripts, the sort of website that can cause an iphone to get red hot, but I think were just autoplaying content and ads) and we found that 22-27 users (note that assumes 100% of people are working which of course they aren't with part-time/flexible working, study, holiday, sickness etc on any given day) on a 32 core machine worked better than 10 on a 16 core as it spread the peaks and troughs more effectively across a larger user pool so I suspect it depends on you specific application stack and what its particular resource utilisation profile looks like, similarly we were able to go from E to D series machines and trade off reducing overprovisioned available RAM to burst into for like 60-90 seconds (but like 8GB at time) for some net extra CPU cores in the pool.

Crossing my fingers for a better day tomorrow

djordan1910 thanks for confirming. My plan tonight is to look to push the one "fairly stable" 32 core VM to 96 cores and 384GB RAM... talk about having all your eggs in one basket but we ended up with nearly 50 users on a D32ads_v5 earlier normally scaled for like 25 and the with machine insights showing <3GB free of 128GB RAM!

I'm currently waiting on Microsoft to pickup the escalated severity A case that apparently 5 hours ago was sat with someone from the "Critical Situation Management and Escalation Team" on why these mitigations don't seem to be working, but at the present rate if ramping that to a single monster machine gets through to the weekend then that will buy time to sort out migrating the data from 4TB of FSLogix profiles into clean new ones.

djordan1910 when you say "For those, we're rebuilding the desktop profile in FSLogix and then they're fine." rebuilding in what way, blowing just the Outlook profile away or blowing the whole FSLogix virtual disk and starting from scratch?

I have to say I was wondering is some of the issue we are seeing still could be broken AppX deployment states in either the machine records (as I think the machine tracks the SIDs for all previous users not just the current ones) or in the FSLogix profile disk somewhere from previous logins as there seem to be not a lot of consistency even on hosts with issues.

PaulGMVP 

It's DISABLED, the MS engineer that sent me this also said to change it to enabled, BUT it's not.. if you read the GPO...