By
Published
Apr 04 2019 05:20 PM
45.5K
Views
First published on TechNet on Apr 08, 2011
Hi folks, Ned here again. Around six years ago we released Service Pack 1 for Windows Server 2003. Like Windows XP SP2, it was a security-focused update. It was the first major server update since the Trustworthy Computing initiative began so there were things like a bootstrapping firewall, Data Execution Protection, and the Security Configuration Wizard.
Amongst all this, the RPC developers added these new configurable group policy settings:
Which map to the DWORD registry settings:
These two settings add an additional authentication "callback capability" to RPC connections. Ordinarily, no authentication is required to make the initial connection to the endpoint mapper (EPM). The EPM is the network service that tells a client what TCP/UDP ports to use in further communications. In Windows, those further communications to the actual application are what typically get authenticated and encrypted. For example, DFSR is an RPC application that uses RPC_C_AUTHN_LEVEL_PKT_PRIVACY with Kerberos required, with Mutual Auth required, and with Impersonation blocked. The EPM connection not requiring authentication is not critical, as there is no application data transmitted: EPM is like a phone book or perhaps more appropriately, a switchboard with an operator.
That quest for Trustworthy Computing added these extra security policies. In doing so, it introduced a very dangerous scenario for domain-based computing: one of the possible policy settings requires all applications that initiate the RPC conversation send along this authentication data or be able to understand a callback request to authenticate.
The problem is most applications have no idea how to satisfy the setting's requirements.
One of the options for Restrictions for unauthenticated RPC clients is "Authenticated without Exceptions".
When enabled, RPC applications are required to authenticate to RPC service on the destination computer. If your application doesn't know how to do this, it is no longer allowed to connect at all.
Which brings us to…
Having configured this policy in your domain on your DCs, members, and clients, you will now see the following issues no matter your credentials or admin rights:
Group policy fails to apply with errors:
Active Directory Replication fails with errors:
Domain join fails with error:
Remotely connecting to WMI returns error:
Remotely connecting to Routing and Remote Access returns error:
Remotely connecting to Disk Management returns error:
Remotely connecting to Component Services (DCOM) returns error:
Running DFSR Health Reports returns errors:
DFSR does not replicate nor start initial sync, with errors:
DFSRMIG does not allow configuration of SYSVOL migration and returns error:
FRS does not replicate and returns event log warning 13562:
Remotely connecting to Windows Firewall with Advanced Security returns error:
Remotely connecting to Share and Storage Management returns error:
Remotely connecting to Storage Explorer returns error:
Remotely connecting to Windows Server Backup returns error:
Remotely connecting to DHCP Management returns error:
RPC Endpoint connections seen through network capture shows errors:
Anyone notice the Catch-22 above? If you deployed this setting using domain-based group policy to your DCs, you have no way to undo it! This is another example of “always test security changes before deploying to production”. Many virtualization products are free, like Hyper-V and Virtual PC – even a single virtualized DC environment would have shown gross problems after you tried to use this policy.
To fix your environment:
These are only the affected Windows in-box applications and components that I have identified. The full list probably includes 99% of all third party RPC applications ever written.
Some security audit consulting company may ask you to turn this policy on to be compliant with their standards. Make sure you show them this article and make them explain why. You can also point out that our Security Compliance Manager tool does not recommend enabling "Authenticated without Exceptions" even in Specialized Security Limited Functionality networks (and SSLF is far too restrictive for most businesses). This setting is really only useful in an unmanaged, standalone, non-domain joined member computer environment such as a DMZ network where you want to close an RPC connection vector. Probably just web servers with local policy.
You should always get in-depth explanation from any third party security audit's findings and recommendations; many a CritSit case here started with a customer implicitly trusting an auditor's recommendations. That auditor is not going to be there to troubleshoot for you when everything goes to crap. Disconnecting all your DCs from the network makes them more secure. So does disabling all your user accounts. Neither is practical.
If you absolutely must turn on Restrictions for unauthenticated RPC clients , make sure it is set only to "Authenticated", and guarantee RPC endpoint mapper client authentication is also enabled . Then test like your job depends on it - because it does. Your applications may still fail with this setting in its less restrictive mode. Not all group policies are intended for domains.
By the way, if you are a software development company you should be giving the Security Development Lifecycle a frank appraisal. It is a completely free force for good.
Until next time.
Ned "2005? I am feeling old" Pyle
Hi folks, Ned here again. Around six years ago we released Service Pack 1 for Windows Server 2003. Like Windows XP SP2, it was a security-focused update. It was the first major server update since the Trustworthy Computing initiative began so there were things like a bootstrapping firewall, Data Execution Protection, and the Security Configuration Wizard.
Amongst all this, the RPC developers added these new configurable group policy settings:
Computer Configuration \ <policies> \ Administrative Templates \ System \ Remote Procedure Call
Restrictions for unauthenticated RPC clients
RPC endpoint mapper client authentication
Which map to the DWORD registry settings:
HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Rpc
RestrictRemoteClients
EnableAuthEpResolution
These two settings add an additional authentication "callback capability" to RPC connections. Ordinarily, no authentication is required to make the initial connection to the endpoint mapper (EPM). The EPM is the network service that tells a client what TCP/UDP ports to use in further communications. In Windows, those further communications to the actual application are what typically get authenticated and encrypted. For example, DFSR is an RPC application that uses RPC_C_AUTHN_LEVEL_PKT_PRIVACY with Kerberos required, with Mutual Auth required, and with Impersonation blocked. The EPM connection not requiring authentication is not critical, as there is no application data transmitted: EPM is like a phone book or perhaps more appropriately, a switchboard with an operator.
That quest for Trustworthy Computing added these extra security policies. In doing so, it introduced a very dangerous scenario for domain-based computing: one of the possible policy settings requires all applications that initiate the RPC conversation send along this authentication data or be able to understand a callback request to authenticate.
The problem is most applications have no idea how to satisfy the setting's requirements.
The Argument
One of the options for Restrictions for unauthenticated RPC clients is "Authenticated without Exceptions".
When enabled, RPC applications are required to authenticate to RPC service on the destination computer. If your application doesn't know how to do this, it is no longer allowed to connect at all.
Which brings us to…
The Brawl
Having configured this policy in your domain on your DCs, members, and clients, you will now see the following issues no matter your credentials or admin rights:
Group policy fails to apply with errors:
GPUPDATE /FORCE returns:
The processing of Group Policy failed. Windows could not resolve the computer name. This could be caused by one of more of the following:
a) Name Resolution failure on the current domain controller.
b) Active Directory Replication Latency (an account created on another domain controller has not replicated to the current domain controller).
Computer Policy update has completed successfully.
To diagnose the failure, review the event log or invoke gpmc.msc to access information about Group Policy results .
The System Event log returns errors 1053 and 1055 for group policy:
The processing of Group Policy failed. Windows could not resolve the user name. This could be caused by one of more of the following:
a) Name Resolution failure on the current domain controller.
b) Active Directory Replication Latency (an account created on another domain controller has not replicated to the current domain controller).
The Group Policy Operational event log will show error 7320:
Error: retrieved account information. Error code 0x5.
Error: Failed to register for connectivity notification. Error code 0x32.
Active Directory Replication fails with errors:
Repadmin.exe returns:
DsBindWithCred to RPC <servername> failed with status 5 (0x5)
DSSites.msc returns:
Directory Service event log returns:
Warning 1655:
Active Directory Domain Services attempted to communicate with the following global catalog and the attempts were unsuccessful.
Global catalog:
\\somedc.cohowineyard.com
The operation in progress might be unable to continue. Active Directory Domain Services will use the domain controller locator to try to find an available global catalog server.
Additional Data
Error value:
5 Access is denied.
Error 1126:
Active Directory Domain Services was unable to establish a connection with the global catalog.
Additional Data
Error value:
1355 The specified domain either does not exist or could not be contacted.
Internal ID:
3200e7b
Warning 2092:
This server is the owner of the following FSMO role, but does not consider it valid. For the partition which contains the FSMO, this server has not replicated successfully with any of its partners since this server has been restarted. Replication errors are preventing validation of this role. Operations which require contacting a FSMO operation master will fail until this condition is corrected.
Domain join fails with error:
Changing the primary domain DNS name of this computer to "" failed.
The name will remain "<something>".
The error was:
Access is denied
After failed join above, rebooting computer and attempting a domain logon fails with error:
The security database on the server does not have a computer account for this workstation trust relationship.
Remotely connecting to WMI returns error:
Win32: Access is denied.
Remotely connecting to Routing and Remote Access returns error:
You do not have sufficient permissions to complete the operation
Remotely connecting to Disk Management returns error:
You do not have access rights to logical disk manager
Remotely connecting to Component Services (DCOM) returns error:
Either the machine does not exist or you don't have permission to access this machine
Running DFSR Health Reports returns errors:
Domain Controller is unreachable
Cannot access the local WMI repository
Cannot connect to reporting DCOM server
DFSR does not replicate nor start initial sync, with errors:
DFSR Event log error 1202:
The DFS Replication service failed to contact domain controller to access configuration information. Replication is stopped. The service will try again during the next configuration polling cycle, which will occur in 60 minutes. This event can be caused by TCP/IP connectivity, firewall, Active Directory Domain Services, or DNS issues.
error: 160 (one or more arguments are not correct)
DFSRMIG does not allow configuration of SYSVOL migration and returns error:
"Unable to connect to the Primary DC's AD. Please make sure that the PDC is reachable and retry the command later"
FRS does not replicate and returns event log warning 13562:
Could not bind to a Domain Controller. Will try again at next polling cycle.
Remotely connecting to Windows Firewall with Advanced Security returns error:
You do not have the correct permissions to open the Windows Firewall with Advanced Security Console.
Error code: 0x5
Remotely connecting to Share and Storage Management returns error:
Connection to the Virtual Disk Service failed. A VDS (Virtual Disk Service) error occurred while performing the requested operation.
Remotely connecting to Storage Explorer returns error:
Access is denied.
Remotely connecting to Windows Server Backup returns error:
The Windows Server Backup engine is not accessible on the computer that you want to manage backups on. Make sure you are a member of the Administrators or Backup Operators group on that computer.
Remotely connecting to DHCP Management returns error:
Access is Denied
RPC Endpoint connections seen through network capture shows errors:
Note how the client (10.90.0.94) attempts to bind to the EPM on a DC (10.90.0.101) and gets rejected with status 0x5 (Access is Denied).
Depending on the calling application - in this case, the Group Policy service running on a Win7 client that is trying to refresh policy - it may continue to try binding many times before giving up. Again, the DC responds with the unhelpful error "REASON_NOT_SPECIFIED" and keeps rejecting the GP service.
For comparison, a normal working EPM bind of the GP service looks like this:
Restitution
Anyone notice the Catch-22 above? If you deployed this setting using domain-based group policy to your DCs, you have no way to undo it! This is another example of “always test security changes before deploying to production”. Many virtualization products are free, like Hyper-V and Virtual PC – even a single virtualized DC environment would have shown gross problems after you tried to use this policy.
To fix your environment:
1. You must delete or unlink the whole policy that includes this RPC setting:
2. Delete or rename this specific policy's GUID folder from each DCs SYSVOL folders (remember, file replication is not working so it must be done on all individual servers).
3. Manually visit all DCs and delete the RestrictRemoteClients registry setting.
4. Reboot all DCs to get your domain back in operation. Not all at once, of course!
These are only the affected Windows in-box applications and components that I have identified. The full list probably includes 99% of all third party RPC applications ever written.
Parole
Some security audit consulting company may ask you to turn this policy on to be compliant with their standards. Make sure you show them this article and make them explain why. You can also point out that our Security Compliance Manager tool does not recommend enabling "Authenticated without Exceptions" even in Specialized Security Limited Functionality networks (and SSLF is far too restrictive for most businesses). This setting is really only useful in an unmanaged, standalone, non-domain joined member computer environment such as a DMZ network where you want to close an RPC connection vector. Probably just web servers with local policy.
You should always get in-depth explanation from any third party security audit's findings and recommendations; many a CritSit case here started with a customer implicitly trusting an auditor's recommendations. That auditor is not going to be there to troubleshoot for you when everything goes to crap. Disconnecting all your DCs from the network makes them more secure. So does disabling all your user accounts. Neither is practical.
If you absolutely must turn on Restrictions for unauthenticated RPC clients , make sure it is set only to "Authenticated", and guarantee RPC endpoint mapper client authentication is also enabled . Then test like your job depends on it - because it does. Your applications may still fail with this setting in its less restrictive mode. Not all group policies are intended for domains.
By the way, if you are a software development company you should be giving the Security Development Lifecycle a frank appraisal. It is a completely free force for good.
Until next time.
Ned "2005? I am feeling old" Pyle
1 Comment