Hello all! This is Chris Cartwright from Directory Services. I had a coworker, Eric Jansen, reach out to me from the field and ask about an incident on site he was looking into a scenario where "the PDCE (Primary Domain Controller Emulator) and DNM (Domain Naming Master) mysteriously moved…" to a DC in another site. He said what was weird was who the logs said performed it. He also said that the other site used their own procedures to build their DCs, which apparently included using Windows Servers Essentials for the base OS. Now, I have never heard of anyone doing that in an enterprise environment, but it got us curious…
Administrators are running the beautiful, pristine Contoso domain forest, (with 2 DCs of course…because two is one, and one is none!). They were built with Windows Server 2012 R2 Datacenter. They know that the FSMOs are still on the first DCs built, and can see that by running "netdom query fsmo":
One day, Contoso-HQ has a new subsidiary, Tailspin Toys that needs Active Directory services for their store. Per Contoso's Organizational Policy, this company will exist on the same domain, and they have authorized admins at that site to promote the new DCs for that site. In preparation for this, the Contoso admins create the site and subnets for the new company's location.
30 days go by and the admins at Contoso notice something odd:
The problem is…nobody authorized that change. They call the admins at Tailspin Toys and asked if they knew anything about this, and they didn’t…. Not good.
So, the first thing that they did was figure out when the change occurred. To do that they used Repadmin.exe with the /ShowObjMeta switch, to see when the FSMORoleOwner attribute changed for the roles that they were interested in. The timestamp for the last modification to that attribute can be found by looking at the object metadata for that attribute at the following DN locations (for each respective role):
- RID Master - "cn=Rid Manager$,cn=system,dc=contoso,dc=com"
- Infrastructure Master - "cn=infrastructure, dc=contoso,dc=com"
- PDC Emulator - "dc=contoso,dc=com"
- Domain Naming Master - "cn=partitions,cn=configuration, dc=contoso,dc=com"
- Schema Master - "cn=schema,cn=configuration, dc=contoso,dc=com"
In this case, they were interested in the Domain Naming Master, and the PDCE Emulator role movements operations.
Once the Contoso Admins figured out when the role was moved and where the change originated from, they now know where to start their search in the event logs. With that said, the Contoso admins transfer the roles back, and then start digging through the logs.
They find these two events in the Directory Services logs on ContosoDC1:
(PDC)
(Domain Naming Master)
So, the change did come from ContosoDC3, but the admins know that they can look at the logs on DC3 to see what user account initiated this, because it's listed with "User."
So, they take a look at ContosoDC3…but what they see isn’t exactly what they expected:
The user is SYSTEM. Has ContosoDC3 become self-aware? Should we expect robots from the future? The administrators continue digging, now focusing on the security logs during the same timeframe. At 8:54:35 PM, the same time that they saw the roles move in the Directory Services log, they now find the following in the security logs on ContosoDC3:
Note the Login ID…
This log entry continued…
This event looks interesting. "An operation was performed on an object." The operation in this case was a "Change PDC Operation" (https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/99dd371d-9ede-417f-beb1-a065c10...)
This just goes to show how important it is to have proper auditing in place...you DO have proper auditing setup, enabled, and verified in your environment, don’t you? If not, I suggest you take a look at Michael Hildebrand’s blog that discusses this in more detail, here.
So, what happened? Well, answering that would have taken a little more logging. Procmon, for example, would have shown that the source port used above came from the silsvc.exe process on the same server.
So, what is this silsvc.exe you ask? That's the Server Infrastructure Licensing Service, and fortunately, it has its own log:
And the final nail in the coffin is the very next event – “The Correction”
To sum everything up, do not put Windows Server Essentials into an existing large enterprise – it was never meant to co-exist, and all of the folks (at least that I talked to in both CSS and PFE) had never seen this scenario before. In the real-world scenario where this happened, those remote ‘spoke site’ DC’s were only meant to be stood up temporarily, and they were. They were stood up just long enough for the compliance threshold to be met, which moved the roles unknowingly to the ‘spoke site’, and the next day those DC’s were taken offline permanently. It just happened that those DC’s were removed on a Friday afternoon and the repercussions weren’t felt until the following week.
We tested this with Server 2012 R2 Essentials and Server 2016 Essentials while manually moving the time forward and saw the same results. Server 2019 Essentials has the same warning, however we were unable to reproduce the issue just by simply moving the time 30 days into the future:
So, it’s being tested the old-fashioned way, and we’ll update on that later in 2021!