Stop Worrying and Love the Outage, Vol IV: Preference items
Note: We apologize for the current viewing experience of these blogs on non-mobile devices. We are working to resolve this issue as soon as possible. This is the fourth article in a series: Stop Worrying and Love the Outage, Vol I: Group Policy and Sharing Violations Stop Worrying and Love the Outage, Vol II: DCs, custom ports, and Firewalls/ACLs Stop Worrying and Love the Outage, Vol III: Cached Logons Hello, Chris Cartwright here from the Directory Services support team, returning (after a brief hiatus) to try to provide the IT community with some tools and verbiage that will hopefully save you and your business many hours, dollars, and frustrations. Today, we will go over something we see sometimes: Group Policy Preference items conflicting with existing client-side extensions (CSE). This scenario can range from going completely unnoticed (or occurring intermittently) to causing full outages. The problem with predicting the impact here is that it is completely dependent on setting in question. Suffice to say, today’s message is, “You should never* target a group policy registry location (including “Windows Settings\Local Policies\Security Options” settings) with a Preference item unless you want instability and an administrative nightmare.” We’ll get to that little disclaimer later. To illustrate this, we will go with Cipher Suite Ordering as an example, since a misconfiguration here can completely stop TLS connections to production services. The setting Here’s a view of gpresult from our example machine: The result This policy setting targets the Functions value seen here in HKLM\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002. Sometimes, it’s this: C:\WINDOWS\system32>reg query HKLM\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002 Functions REG_SZ TLS_AES_256_GCM_SHA384 Sometimes…it’s this: C:\WINDOWS\system32>reg query HKLM\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002 Functions REG_SZ TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_NULL_SHA256,TLS_RSA_WITH_NULL_SHA,TLS_PSK_WITH_AES_256_GCM_SHA384,TLS_PSK_WITH_AES_128_GCM_SHA256,TLS_PSK_WITH_AES_256_CBC_SHA384,TLS_PSK_WITH_AES_128_CBC_SHA256,TLS_PSK_WITH_NULL_SHA384,TLS_PSK_WITH_NULL_SHA256 These are some different Data for the same registry value. So…why is this happening? Well, if we had looked further down: Here, we have two different GPOs targeting the same registry key. One, correctly uses an Administrative Template to modify a registry value under a SOFTWARE\Policies key. The other, a Preference item, inappropriatelytargets the same policy key. Why “inappropriately”? Because you should never* target a group policy registry location (including “Windows Settings\Local Policies\Security Options” settings) with a Preference item unless you want instability. Now, that we’ve gotten that message out of the way... the weirdness: This is a Procmon of several gpupdate commands. We see group policy behaving as expected here three times (seen in blue), calling the Registry CSE first (think Administrative Templates), writing some values and data, and then the Group Policy Registry CSE (think Registry preference items) comes along and replaces it. This is the order that these extensions are processed in. At this point, who could tell what the intended data of this value was to begin with? Anyway, it does this three times (because I told it to refresh three times) and then doesn’t do it on the fourth refresh (single line in red). Instead, the Registry CSE applies and nothing else happens. Why? To answer that, we need to turn on Group Policy debug logging. Here is where Preference Items “wins” (significantly trimmed): GPSVC(28c.fe0) 12:46:25:697 ProcessGPOs(Machine): ----------------------- GPSVC(28c.fe0) 12:46:25:697 ProcessGPOs(Machine): Processing extension Registry This is Admin Template Land GPSVC(28c.fe0) 12:46:25:900 ParseRegistryFile: Entering with <\\contoso.com\SysVol\contoso.com\Policies\{9804AF19-19F6-4C96-AEB2-B97DD318F123}\Machine\registry.pol>. GPSVC(28c.fe0) 12:46:25:916 SetRegistryValue: Functions => TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, <snip>... [OK] GPSVC(28c.fe0) 12:46:25:916 ParseRegistryFile: Leaving. GPSVC(28c.fe0) 12:46:27:494 ProcessGPOs(Machine): Processing extension Group Policy Registry This is Preference Item Land GPSVC(28c.fe0) 12:46:27:494 ReadStatus: Read Extension's Previous status successfully. GPSVC(28c.fe0) 12:46:27:494 ReadGPOList:++ GPSVC(28c.fe0) 12:46:27:494 ReadGPOList: Read Key:0 GPSVC(28c.fe0) 12:46:27:494 ReadGPOList: Read Key:1 GPSVC(28c.fe0) 12:46:27:494 ReadGPOList:-- (Result:TRUE) GPSVC(28c.fe0) 12:46:27:494 CheckGPOs: ReadGPOList count = 2 GPSVC(28c.fe0) 12:46:27:494 CompareGPOLists: The lists are the same. GPSVC(28c.fe0) 12:46:27:494 CheckGPOs: No GPO changes but called in force refresh flag or extension Group Policy Registry needs to run force refresh in foreground processing GPSVC(28c.fe0) 12:46:27:494 GPLockPolicySection: Sid = (null), dwTimeout = 30000, dwFlags = 0x40 GPSVC(28c.fe0) 12:46:27:494 bMachine = 1 GPSVC(28c.fe0) 12:46:27:494 Global Sync Lock Called GPSVC(28c.fe0) 12:46:27:494 Writer Lock got immediately. GPSVC(28c.fe0) 12:46:27:494 Global Lock taken successfully GPSVC(28c.fe0) 12:46:27:494 ProcessGPOList:++ Entering for extension Group Policy Registry GPSVC(28c.fe0) 12:46:27:494 ProcessGPOList: Passing in the force refresh flag to Extension Group Policy Registry GPSVC(28c.fe0) 12:46:27:494 LogExtSessionStatus: Successfully logged Extension Session data GPSVC(28c.fe0) 12:46:27:494 ProcessGPOList: lpGPOInfo->lpGPInfoHandle->dwExtnCount is 2 for Group Policy Registry. GPSVC(28c.fe0) 12:46:27:838 ProcessGPOList: Extension Group Policy Registry returned 0x0. The above shows that the Administrative Template CSE executes first, and then Preference Items comes along. And here is where the Administrative Template “wins” instead (significantly trimmed): GPSVC(28c.fe0) 12:48:47:351 ProcessGPOs(Machine): ----------------------- GPSVC(28c.fe0) 12:48:47:351 ProcessGPOs(Machine): Processing extension Registry This is Admin Template Land GPSVC(28c.fe0) 12:48:47:538 ParseRegistryFile: Entering with <\\contoso.com\SysVol\contoso.com\Policies\{9804AF19-19F6-4C96-AEB2-B97DD318F123}\Machine\registry.pol>. GPSVC(28c.fe0) 12:48:47:554 SetRegistryValue: Functions => TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, <snip> ...[OK] GPSVC(28c.fe0) 12:48:47:554 ParseRegistryFile: Leaving. ~~~~~~~~~~~Large jump here~~~~~~~ GPSVC(28c.fe0) 12:48:48:038 ProcessGPOs(Machine): Processing extension Group Policy Registry This is Preference Item Land GPSVC(28c.fe0) 12:48:48:038 ReadStatus: Read Extension's Previous status successfully. GPSVC(28c.fe0) 12:48:48:054 ReadGPOList:++ GPSVC(28c.fe0) 12:48:48:054 ReadGPOList: Read Key:0 GPSVC(28c.fe0) 12:48:48:054 ReadGPOList: Read Key:1 GPSVC(28c.fe0) 12:48:48:054 ReadGPOList:-- (Result:TRUE) GPSVC(28c.fe0) 12:48:48:054 CheckGPOs: ReadGPOList count = 2 GPSVC(28c.fe0) 12:48:48:054 CompareGPOLists: The lists are the same. GPSVC(28c.fe0) 12:48:48:054 CheckGPOs: No GPO changes and no security group membership change and extension Group Policy Registry has NoGPOChanges set. In this one, Registry runs, sets the value, and then Group Policy Registry starts and then says, “No GPO changes and no security group membership change and extension Group Policy Registry has NoGPOChanges set.” So, what does that mean? I managed to salvage this from the ill-advised archival of Windows Server 2003 documentation: NoGPOListChanges is a registry value that can be configured for CSEs. When NoGPOListChanges is set to 0, “GPO Changes” above effectively is always yes. Here, we can see the registry value is configured to be 1 for the Group Policy Registry CSE: This means that if no changes are detected, we will skip processing. That came from this GPO (Thank you, Procmon): Thus ends the mystery of the intermittency. Since the Force flag wasn’t set, and NoGPOListChanges is not enabled, the Preference value doesn’t get applied this time. Remember, I’m not advising you how to make this work. By default, Group Policy Preferences would always reapply anyway unless ‘Apply once and do not reapply’ is enabled. I am just pointing out how you can have an unstable configuration by deploying preference items that you shouldn’t. In addition to this technical issue above, think about the administrative implications. Do you really want to have to constantly compare your list of 300 preference items to your admin templates every time you want to make a change? Friends don’t let friends target preference items at group policy settings. -Chris “Prefer no conflict” Cartwright * Okay. I did promise to talk about the disclaimer later. There are character limits to some settings. For example, the setting I used here, SSL Cipher Suite Order has a character limit of 1,023. In earlier, unsupported versions of Windows, it was not only possible, but likely, to have a list that went past this limit. In this case, the only Group Policy recourse would be to use this preference item. However, if you were to do this, you would also make sure that SSL Cipher Suite Order was not configured in Admin templates. Given that this is no longer a problem in supported OSes, it is not a valid excuse. I am unaware of any similar settings that exist on a supported OS at this time. References: https://learn.microsoft.com/en-us/previous-versions/windows/desktop/policy/creating-a-policy-callback-function https://techcommunity.microsoft.com/t5/core-infrastructure-and-security/hey-dude-where-s-my-winlogon-log/ba-p/259042
