Hi, Linda Taylor here, Senior Escalation Engineer from the Directory Services team in the UK. I have been working on this issue which seems to be affecting many of you globally on windows 8.1, 2012 R2 and windows 10, so I thought it would be a good idea to explain the issue and workarounds while we continue to work on a proper fix here. The symptoms are such that after a password change, logon hangs forever on the welcome screen:
How annoying…. The underlying issue is a deadlock between several components including DPAPI and the redirector. Why does it happen? So far we have seen this issue in the following circumstances (always after a password change/reset which is done somewhere other than the users machine – i.e. on the DC or in a portal) 1. If the user has a home drive which maps to a DFS like path for example: \\contoso.com\homefolders\user1 OR 2. If The following GPO is applied: Computer configuration\administrative templates Windows Components\File Explorer\ “Set a default associations configuration file” And the XML file is stored on a DFS based path. For example \\contoso.com\netlogon OR 3. If you use GPP to map drives during logon to a DFS path like \\contoso.com\someShare This issue happens due to a deadlock between DPAPI, Credential manager and the Redirector (RDR). It goes like this… 1. When the user logs on, the profile service tries to map network home folder to \\contoso.com\... 2. To do this, we need to have a call created in RDR, and this requires a SMB session setup to dcname.contoso.com 3. The SMB session setup requires a security blob created to authenticate with the target server, which is the DC. 4. To create the security blob, Kerberos will check saved credentials by calling DPAPI. 5. DPAPI cannot decode the saved credential because the master key is not available because the user's password is reset on DC, so it will need to query the DC for a master key. This requires a named pipe call to \\dcname.contoso.com\IPC$\protected_storage 6. To connect to this named pipe, RDR found it is the same as previous call in#2 (same fqdn DC name \\dcname.contoso.com) so now session setup is queued… 7. The Kerberos thread will hang forever, and the profile service will hang forever until a reboot. 8. After reboot, the user still cannot logon with the same symptom. (note: a different user CAN log on). The problem occurs on client computers with Windows 8.1, Windows 10 and also Windows Server 2012 R2 (for example RDS scenario). The problem occurs most frequently after an admin password reset which has occurred elsewhere (not the on the client computer to which the logon is happening) but it can also occur when the password change is not recent if the user is logging onto a machine where the cached credentials are old and they have changed their password on some other machine some time ago.
So, what can you do to get out of this problem? There are several options for working around the issue:
1. If you have the mentioned policy – move the XML to some file share which is not DFS based and is not on a DC.
2. Assuming no-one wants to change home drive paths because there are many users and it’s a hassle, the other option is to disable Credential manager and clear the cached credentials. When there are no entries for credential manager, there are no reasons to access the DC to refresh the keys. Hence the dead-lock. This could be done by disabling this policy: https://technet.microsoft.com/en-us/library/jj852185.aspx
3. Dig your way out by connecting to the machine remotely and deleting the entry under C:\Users\<username>\AppData\Roaming\Microsoft\Protect\[Problem Users SID]. Note if there is a roaming profile you can also delete this entry on the profile server and it works. This is because the profile is downloaded before the drive mapping takes place.
4. Another (not so great) option is to boot the PC without any network cable and log on with the old password. Then connect it back.
This issue also has a long history and there are other variant’s of this deadlock which were fixed before. See below list of related fixes:
Related previous Fixes for Windows 8.1 an Windows 2012 R2:
Finally, Microsoft is actively working on fixes for Windows 8.1 / WS 2012 R2 and Windows 10 TH2 for the current issue described above. I will share the release dates and article #’s once known. If you experience this issue please ensure you have all of the above fixes in place and use the workarounds noted above and keep an eye out on updates to this blog.