First published on TechNet on Jan 11, 2016
********** UPDATE ********** This is now fixed in the following updates: For Windows 8.1, 2012 R2, 2012 install:
- KB3132080 Logon freezes after you reset your password in Windows 8.1, or Stop error 0x1000007e in Windows Server 2012 R2: http://support.microsoft.com/kb/3132080/EN-US
For Windows 10 TH2 build 1511 install:
- KB3135173 Cumulative update for Windows 10 Version 1511: February 9, 2016: http://support.microsoft.com/kb/3135173/EN-US
******************************
Hi, Linda Taylor here, Senior Escalation Engineer from the Directory Services team in the UK. I have been working on this issue which seems to be affecting many of you globally on windows 8.1, 2012 R2 and windows 10, so I thought it would be a good idea to explain the issue and workarounds while we continue to work on a proper fix here. The symptoms are such that after a password change, logon hangs forever on the welcome screen:
How annoying…. The underlying issue is a deadlock between several components including DPAPI and the redirector. Why does it happen? So far we have seen this issue in the following circumstances (always after a password change/reset which is done somewhere other than the users machine – i.e. on the DC or in a portal) 1. If the user has a home drive which maps to a DFS like path for example: \\contoso.com\homefolders\user1 OR 2. If The following GPO is applied: Computer configuration\administrative templates Windows Components\File Explorer\ “Set a default associations configuration file” And the XML file is stored on a DFS based path. For example \\contoso.com\netlogon OR 3. If you use GPP to map drives during logon to a DFS path like \\contoso.com\someShare This issue happens due to a deadlock between DPAPI, Credential manager and the Redirector (RDR). It goes like this… 1. When the user logs on, the profile service tries to map network home folder to \\contoso.com\... 2. To do this, we need to have a call created in RDR, and this requires a SMB session setup to dcname.contoso.com 3. The SMB session setup requires a security blob created to authenticate with the target server, which is the DC. 4. To create the security blob, Kerberos will check saved credentials by calling DPAPI. 5. DPAPI cannot decode the saved credential because the master key is not available because the user's password is reset on DC, so it will need to query the DC for a master key. This requires a named pipe call to \\dcname.contoso.com\IPC$\protected_storage 6. To connect to this named pipe, RDR found it is the same as previous call in#2 (same fqdn DC name \\dcname.contoso.com) so now session setup is queued… 7. The Kerberos thread will hang forever, and the profile service will hang forever until a reboot. 8. After reboot, the user still cannot logon with the same symptom. (note: a different user CAN log on). The problem occurs on client computers with Windows 8.1, Windows 10 and also Windows Server 2012 R2 (for example RDS scenario). The problem occurs most frequently after an admin password reset which has occurred elsewhere (not the on the client computer to which the logon is happening) but it can also occur when the password change is not recent if the user is logging onto a machine where the cached credentials are old and they have changed their password on some other machine some time ago.
So, what can you do to get out of this problem? There are several options for working around the issue:
1. If you have the mentioned policy – move the XML to some file share which is not DFS based and is not on a DC.
2. Assuming no-one wants to change home drive paths because there are many users and it’s a hassle, the other option is to disable Credential manager and clear the cached credentials. When there are no entries for credential manager, there are no reasons to access the DC to refresh the keys. Hence the dead-lock. This could be done by disabling this policy: https://technet.microsoft.com/en-us/library/jj852185.aspx
3. Dig your way out by connecting to the machine remotely and deleting the entry under C:\Users\<username>\AppData\Roaming\Microsoft\Protect\[Problem Users SID]. Note if there is a roaming profile you can also delete this entry on the profile server and it works. This is because the profile is downloaded before the drive mapping takes place.
4. Another (not so great) option is to boot the PC without any network cable and log on with the old password. Then connect it back.
This issue also has a long history and there are other variant’s of this deadlock which were fixed before. See below list of related fixes:
Related previous Fixes for Windows 8.1 an Windows 2012 R2:
| Windows 8.1 & 2012 R2 KB# | X64 file versions for dpapisrv.dll and lsasrv.dll | |||||
| https://support.microsoft.com/en-us/kb/3101183 “ You can't log on to a domain-joined computer in Windows 8.1 or Windows Server 2012 R2” Released in October 2015. Note: this is the latest but does not resolve this variant of the issue. | DPAPISRV.dll 6.3.9600.18088. LSASRV.dll version 6.3.9600.18088. | |||||
| 3038562 Cannot access DPAPI data after an administrator resets your password on a Windows Server 2012 R2-based domain controller http://support.microsoft.com/kb/3038562/EN-US |
|
|||||
| The pre-requisite to 3101183 and 3038562 is April update: 2919355 Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 Update: April 2014 http://support.microsoft.com/kb/2919355/EN-US This contains win8.1 version of KB2927267 https://support.microsoft.com/en-us/kb/2927 267 “ You cannot log on to Windows after the admin changes your password” | Updates LSASRV.dll 6.3.9600.17042 |
There are no previous fixes for Windows 10.
There are some related past fixes for similar deadlocks in windows 8 also:
| Windows 8 KB# | X64 file versions for dpapisrv.dll and lsasrv.dll |
| 3084956 You can‘t log on to a domain-joined computer in Windows 8 or Windows Server 2012 http://support.microsoft.com/kb/3084956/EN-US - Released September. (NOTE: This is the windows 8 equivalent of KB3101183.) | X64 file versions: DPAPISRV.dll 6.2.9200.21645 and LSASRV.dll Version 6.2.9200.21582 |
| KB3049843 https://support.microsoft.com/en-us/kb/3049843 - You cannot access DPAPI data after an administrator resets your password on a Windows Server 2012-based domain controller | Dpapisrv.dll 6.2.9200.21442 |
| 2927267: https://support.microsoft.com/en-us/kb/2927267 | Lsasrv.dll 6.2.9200.20931 |
Finally, Microsoft is actively working on fixes for Windows 8.1 / WS 2012 R2 and Windows 10 TH2 for the current issue described above. I will share the release dates and article #’s once known. If you experience this issue please ensure you have all of the above fixes in place and use the workarounds noted above and keep an eye out on updates to this blog.
Linda Taylor