First published on TechNet on Apr 16, 2012
Hello! Jake Mowrer here to share an “experience” with you all that will hopefully help you avoid running into a nightmare the next time you add a DFS Namespace (DFS-N) server. I find that DFS-N in customer environments works well, so well that they require very little maintenance. This tends to lead to admins not wanting to touch the environment unless they really have to, which in turn leads to “We don’t look directly into the eyes of that system, it just works, not sure how and we’re not sure who manages it.”
Ultimately, you will have to add systems to host DFS-N whether due to an OS upgrade, a new AD site rolling out, or just need to scale out a bit. This is what my client and I were doing one late Tuesday night which seemed to go well, until the calls started pouring into the helpdesk the next morning.
We were adding a Windows Server 2008 R2 server into the domain based DFS-N which was entirely Windows Server 2003. The reason we were making this change was that a site was complaining about slow DFS Namespace enumeration. Upon further investigation, we found this site had to hop over two sites to get to the nearest DFS-N. So easy enough, we figured we would add a DFS-N server in a closer site to improve the enumeration time. Here were the steps we used:
- From a Windows 7 client with the DFS File System RSAT tools installed or a Windows XP machine with the Windows XP Support Tools installed, we ran a DFSUTIL /PKTINFO to confirm what server the client was using for the DFS-N prior to the change. Here was the relevant output:
- Using the DFS console on the newly installed Windows Server 2008 R2 machine, we added the namespace to the new DFS-N server:
- We clicked Edit Settings to make sure All Users had read to the share.
- Clicked OK and it added successfully.
- We then did a DFSUTIL /PKTFLUSH on the same client, accessed the namespace, then ran a DFSUTIL /PKTINFO, and everything looked fine:
- The results showed it came up in 2 seconds vs. 20 seconds, so we were happy.
We all went to sleep.
Winter is coming, and so are the helpdesk tickets.
I woke up to my phone ringing and it was the morning shift admin indicating there was a DFS namespace issue. The symptom was that Windows XP clients were receiving this error when accessing a subfolder in the DFS Namespace:
“This file does not have a program associated with it for performing this action. Create an association in the Folder Options control panel.”
“What the heck kind of error is that when trying to open a folder?!?!” I thought to myself. Here are the things we checked:
- Ensured the server name was entered correctly in the DFS console, it was.
- Ran another pktflush on the client, didn’t resolve.
- Ensure the share existed on the serveralotcloser machine, it did.
By this time it was getting about an hour into the outage, so we were running out of time to find root cause, eventually we needed to back out the change.
Before we did, I had the admin capture a network trace using Network Monitor 3.4 (shameless plug) and repro the issue. We removed the new server from the namespace and the problem was resolved.
But how are we going to get these servers added with this issue lingering out there?
I took some time to wake up, eat a Cliff bar, and drive into their office. I looked over the network trace and here is what caught my eye:
That looks like a permissions issue, but I checked the share permissions, so it had to be NTFS!
Researching cases I found one case where my friend David Everett from our DS support team used iCACLS to look at the permissions.
I ran the iCACLS command against the existing servers in the DFS-N (ie. The working servers) and here is what they looked like:
BUILTIN\Administrators:(OI)(CI)(F)
NT AUTHORITY\SYSTEM:(OI)(CI)(F)
BUILTIN\Administrators:(F)
CREATOR OWNER:(OI)(CI)(IO)(F)
BUILTIN\Users:(OI)(CI)(RX)
BUILTIN\Users:(CI)(AD)
BUILTIN\Users:(CI)(WD)
I saw his comment in the case that if the permissions were inherited they wouldn’t replicate.
We checked the c:\dfsroots directory on the server that we removed (it remained there even after removing the server from the DFS-N configuration) and here is what showed up:
BUILTIN\Administrators:(OI)(CI)(F)
NT AUTHORITY\SYSTEM:(OI)(CI)(F)
BUILTIN\Administrators:(F)
CREATOR OWNER:(OI)(CI)(IO)(F)
Again all inherited but notice that the BUILTIN\Users group is not listed. Where were these permissions coming from, we never set them when adding the server to the DFS-N root? Why was BUILTIN\Users missing?
<Pause for suspense>
They were coming down from the root of the C: drive, meaning the customer had changed the default permissions on the C: drive in their build process. This filtered down to the c:\dfsroots directory thus “softly” denying “Users” access to the folder. By default on Windows Server 2008 R2, BUILTIN\Users (or computername\Users if you are not on a DC) should have Read/Execute/List Folders on the C: drive. To harden security, the customer removed this ACE on their Windows Server 2008 R2 build.
To fix this on the server we planned on adding to the DFS-N root, we added the BUILTIN\Users group and gave it Read/Execute to the C:\dfsroots directory. We could have added it back to the root of the C: drive but I didn’t want to make that big of a change. Once we added the server back into the DFS-N root we tested using the same method above and it worked as expected. Problem resolved!
To wrap things up:
I hear all the time that hardening a server also adds complexity, this is one prime example.
I asked myself, “So what should we have done different to make the change less impactful?” Here’s my “should have” list:
- We should have tested with an account that had equal privileges to one that was actually used in the field to access the namespace. I thought we were, but I never asked the customer to confirm, shame on me.
- We should have checked the NTFS permissions for c:\dfsroots on the new DFS-N root servers after making the change.
I hope this write up helps add one more thing to your checklist when adding a server to your DFS Namespace. Have a great week!
Jake Mowrer