Friday Mail Sack: Super Slo-Mo Edition

[Ned Pyle]

First published on TechNet on Sep 30, 2011
Hello folks, Ned here again with another Mail Sack. Before I get rolling though, a quick public service announcement:

Plenty of you have downloaded the Windows 8 Developer Preview and are knee-deep in the new goo . We really want your feedback, so if you have comments, please use one of the following avenues:

• Post on the MSDN Win8 General Forum
  This one is watched by the Product Groups, so any feedback here goes right to the horse's, ehhh, mouth. :) This is the preferred approach.


• Email AskDS, using our contact page
  I will personally ensure the appropriate developers read your feedback if you email it to us (as long as it is not crazy, natch).

I recommend sticking to IT Pro features; the consumer side’s covered and the biggest value is your Administrator experience. The NDA is not off - I still cannot comment on the future of Windows 8 or tell you if we already have plans to do X with Y. This is a one-way channel from you to us (to the developers).

Cool? On to the sack. This week we discuss:

• How password changes are protected on the network


• DISM Destroying DCs


• Locked out guest account


• Changing the NDES service password


• If you really need a GC


• DFSR and symbolic links


• Increase your memory (not what you're thinking)


• Other stuff

Shake it.

Question

We were chatting here about password synchronization tools that capture password changes on a DC and send the clear text password to some third party app. I consider that a security risk...but then someone asked me how the password is transmitted between a domain member workstation and a domain controller when the user performs a normal password change operation (CTRL+ALT+DEL and Change Password). I suppose the client uses some RPC connection, but it would be great if you could point me to a reference.

Answer

Windows can change passwords many ways - it depends on the OS and the component in question.

1. For the specific case of using CTRL+ALT+DEL because your password has expired or you just felt like changing your password:

If you are using a modern OS like Windows 7 with AD, the computer uses the Kerberos protocol end to end. This starts with a normal AS_REQ logon, but to a special service principal name of kadmin/changepw , as described in http://www.ietf.org/rfc/rfc3244.txt .

The computer first contacts a KDC over port 88, then communicates over port 464 to send along the special AP_REQ and AP_REP. You are still using Kerberos cryptography and sending an encrypted payload containing a KRB_PRIV message with the password. Therefore, to get to the password, you have to defeat Kerberos cryptography itself, which means defeating the crypto and defeating the key derived from the cryptographic hash of the user's original password. Which has never happened in the history of Kerberos.

The parsing of this kpasswd traffic is currently broken in NetMon's latest public parsers, but even when you parse it in WireShark, all you can see is the encryption type and a payload of encrypted goo. For example, here is that Windows 7 client talking to a Windows Server 2008 R2 DC, which means AES-256:

Aka: Insane-O-Cryption ™

On the other hand, if using a crusty OS like Windows XP, you end up using a legacy password mechanism that worked with NT 4.0 – in this case SamrUnicodeChangePasswordUser2 ( http://msdn.microsoft.com/en-us/library/cc245708(v=PROT.10).aspx) .

XP also supports the Kerberos change mechanism, but by default uses NTLM with CTRL+ALT+DEL password changes. Witness:

This uses “RPC over SMB with Named Pipes” with RPC packet privacy. You are using NTLM v2 by default (unless you set LMCompatibility unwisely) and you are still double-protected (the payload and packets), which makes it relatively safe. Definitely not as safe as Win7 though – just another reason to move forward.

You can disable NTLM in the domain if you have Win2008 R2 DCs and XP is smart enough to switch to using Kerberos here:

... but you are likely to break many other apps . Better to get rid of Windows XP.

2. A lot of administrative code use SamrSetInformationUser2, which does not require knowing the user’s current password ( http://msdn.microsoft.com/en-us/library/cc245793(v=PROT.10).aspx ). For example, when you use NET USER to change a domain user’s password:

This invokes SamrSetInformationUser2 to set Internal4InformationNew data:

So, doubly-protected (a cryptographically generated, key signed hash covered by an encrypted payload). This is also “RPC over SMB using Named Pipes”

The crypto for the encrypted payload is derived from a key signed using the underlying authentication protocol, seen from a previous session setup frame (negotiated as Kerberos in this case):

3. The legacy mechanisms to change a user password are NetUserChangePassword ( http://msdn.microsoft.com/en-us/library/windows/desktop/aa370650(v=vs.85).aspx ) and IADsUser::ChangePassword ( http://msdn.microsoft.com/en-us/library/windows/desktop/aa746341(v=vs.85).aspx )

4. A local user password change usually involves SamrUnicodeChangePasswordUser2, SamrChangePasswordUser, or SamrOemChangePasswordUser2 ( http://msdn.microsoft.com/en-us/library/cc245705(v=PROT.10).aspx ).

There are other ways but those are mostly corner-case.

Note: In my examples, I am using the most up to date Netmon 3.4 parsers from http://nmparsers.codeplex.com/ .

Question

If I try to remove the AD Domain Services role using ServerManager.msc , it blocks me with this message:

But if I remove the role using Dism.exe , it lets me continue:

This completely hoses the DC and it no longer boots normally. Is this a bug?

And - hypothetically speaking, of course - how would I fix this DC?

Answer

Don’t do that. :)

Not a bug, this is expected behavior. Dism.exe is a pure servicing tool; it knows nothing more of DCs than the Format command does. ServerManager and servermanagercmd.exe are the tools that know what they are doing.
Update: Although as Artem points out in the comments, we want you to use the Server Manager PowerShell and not servermanagercmd, which is on its way out.

To fix your server, pick one:

• Boot it into DS Repair Mode with F8 and restore your system state non-authoritatively from backup (you can also perform a bare metal restore if you have that capability - no functional difference in this case). If you do not have a backup and this is your only DC, update your résumé.
  


• Boot it into DS Repair Mode with F8 and use dcpromo /forceremoval to finish what you started. Then perform metadata cleanup . Then go stand in the corner and think about what you did, young man!

Question

We are getting Event ID 4740s ( account lockout ) for the AD Guest account throughout the day, which is raising alerts in our audit system. The Guest account is disabled, expired, and even renamed. Yet various clients keep locking out the account and creating the 4740 event. I believe I've traced it back to the occasional attempt of a local account attempting to authenticate to the domain. Any thoughts?

Answer

You'll see that when someone has set a complex password on the Guest account, using NET USER for example, rather than having it be the null default. The clients never know what the guest password is, they always assume it's null like default - so if you set a password on it, they will fail. Fail enough and you lock out (unless you turn that policy off and replace it with intrusion protection detection and two-factor auth). Set it back to null and you should be ok. As you suspected, there a number of times when Guest is used as part of a "well, let's try that " algorithm:

Network access validation algorithms and examples for Windows Server 2003, Windows XP, and Wind...

To set it back you just use the Reset Password menu in Dsa.msc on the guest account, making sure not to set a password and clicking ok. You may have to adjust your domain password policy temporarily to allow this.

As for why it's "locking out" even though it's disabled and renamed:

• It has a well-known SID (S-1-5-21- domain -501) so renaming doesn’t really do anything except tick a checkbox on some auditor's clipboard


• Disabled accounts can still lock out if you keep sending bad passwords to them. Usually no one notices though, and most people are more concerned about the "account is disabled" message they see first.

Question

What are the steps to change the "User Account" password set when the Network Device Enrollment Service (NDES) is installed?

Answer

When you first install the Network Device Enrollment Service (NDES), you have the option of setting the identity under which the application pool runs to the default application pool identity or to a specific user account. I assume that you selected the latter. The process to change the password for this user account requires two steps -- with 27 parts (not really…).

1. First, you must reset the user account's password in Active Directory Users and Computers.

2. Next, you must change the password configured in the application pool Advanced Settings on the NDES server.

a. In IIS manager, expand the server name node.

b. Click on Application Pools .

c. On the right, locate and highlight the SCEP application pool.

d. In the Action pane on the right, click on Advanced Settings....

e. Under Process Model click on Identity, then click on the … button.



f. In the Application Pool Identity dialog box, select Custom account and then click on Set….

g. Enter the custom application pool account name, and then set and confirm the password. Click Ok, when finished.



h. Click Ok, and then click Ok again.

i. Back on the Application Pools page, verify that SCEP is still highlighted. In the Action pane on the right, click on Recycle….

j. You are done.

Normally, you would have to be concerned with simply resetting the password for any service account to which any digital certificates have been assigned. This is because resetting the password can result in the account losing access to the private keys associated with those certificates. In the case of NDES, however, the certificates used by the NDES service are actually stored in the local computer's Personal store and the custom application pool identity only has read access to those keys. Resetting the password of the custom application pool account will have no impact on the master key used to protect the NDES private keys.

[Courtesy of Jonathan , naturally - Neditor]

Question

If I have only one domain in my forest, do I need a Global Catalog? Plenty of documents imply this is the case.

Answer

All those documents saying "multi-domain only" are mistaken. You need GCs - even in a single-domain forest - for the following:

( Update: Correction on single-domain forest logon made, thanks for catching that Yusuf! I also added a few more breakage scenarios)

• Perversely, if you have enabled IgnoreGCFailures ( http://support.microsoft.com/kb/241789 ); turning it on removes universal groups from the user security token if there is no GC, meaning they will logon but not be able to access resources they accessed fine previously).


• If your users logon with UPNs and try to change their password (they can still logon in a single domain forest with UPN or NetBiosDomain\SamAccountName style logons).


• Even if you use Universal Group Membership Caching to avoid the need for a GC in a site, that DC needs a GC to update the cache.


• MS Exchange is deployed (All versions of Exchange services won't even start without a GC).


• Using the built-in Find in the shell to search AD for published shares, published DFS links, published printers, or any object picker dialog that provides option "entire directory"  will fail.


• DPM agent installation will fail.


• AD Web Services (aka AD Management Gateway) will fail.


• CRM searches will fail.


• Probably other third parties of which I'm not aware.

We stopped recommending that customers use only handfuls of GCs years ago - if you get an ADRAP or call MS support, we will recommend you make all DCs GCs, unless you have an excellent reason not. Our BPA tool states that you should have at least one GC per AD site : http://technet.microsoft.com/en-us/library/dd723676(WS.10).aspx .

Question

If I use DFSR to replicate a folder containing symbolic links , will this replicate the source files or the actual symlinks? The DFSR FAQ says symlink replication is supported under certain circumstances.

Answer

The symlink replicates; however, the underlying data does not replicate just because there is a symlink. If the data is not stored within the RF, you end up with a replicated symlink to nowhere:

Server 1, replicating a folder called c:\unfiltersub. Note how the symlink points to a file that is not in the scope of replication:

Server 2, the symlink has replicated - but naturally, it points to an un-replicated file. Boom:

If the source data is itself replicated, you’re fine. There’s no real way to guarantee that though, except preventing users from creating files outside the RF by using permissions and FSRM screens. If your end users can only access the data through a share, they are in good shape. I'd imagine they are not the ones creating symlinks though. ;-)

Question

I read your post on career development . There are many memory techniques and I know everyone is different, but what do you use?

[A number of folks asked this question - Neditor]

Answer

When I was younger, it just worked - if I was interested in it, I remembered it. As I get older and burn more brain cells though, I find that my best memory techniques are:

• Periodic skim and refresh. When I have learned something through deep reading and hands on, I try to skim through core topics at least once a year. For example, I force myself to scan the diagrams in the all the Win2003 Technical Reference A-Z sections , and if I can’t remember what the diagram is saying, I make myself read that section in detail. I don’t let myself get too stale on anything and try to jog it often.


• Mix up the media. When learning a topic, I read, find illustrations, and watch movies and demos. When there are no illustrations, I use Visio to make them for myself based on reading. When there are no movies, I make myself demo the topics. My brain seems to retain more info when I hit it with different styles on the same subject.


• I teach and publically write about things a lot . Nothing hones your memory like trying to share info with strangers, as the last thing I want is look like a dope. It makes me prepare and check my work carefully, and that natural repetition – rather than forced “read flash cards”-style repetition, really works for me. My brain runs best under pressure.


• Your body is not a temple (of Gozer worshipers) . Something of a cliché, but I gobble vitamins, eat plenty of brain foods , and work out at least 30 minutes every morning.

I hope this helps and isn’t too general. It’s just what works for me.

Other Stuff

Have $150,000 to spend on a camera, a clever director who likes FPS gaming, and some very fit paint ballers? Go make a movie better than this . Watch it multiple times.

Once for the chat log alone

Best all-around coverage of the Frankfurt Auto Show here, thanks to Jalopnik .

Want!

The supposedly 10 Coolest Death Scenes in Science Fiction History . But any list not including Hudson’s last moments in Aliens is fail.

If it’s true… holy crap! Ok, maybe it wasn’t true. Wait, HOLY CRAP !

So many awesome things combined .

Finally, my new favorite time waster is Retronaut . How can you not like a website with things like “Celebrities as Russian Generals”.

No, really.

Have a nice weekend folks,

- Ned “Oh you want some of this?!?!” Pyle