customsd controls event log ChannelAccess and causing IIS ARR module unable to register
Published Apr 17 2024 04:54 PM 1,814 Views
Microsoft

Introduction of issue symptom:

1. With ARR module in IIS global module list, w3wp.exe got 503 at first request - the w3wp.exe process crash consistently.

2. If remove ARR RequestRouter.dll from IIS global module list, then w3wp is up no issue.

3. The issue is only on servers in one domain - Could be domain GPOs caused the problem.

 

Time Travel Dump pinned the failed code path in the call stack below:

00 (Inline Function) --------`-------- rpcrt4!GENERIC_OBJECT::InvalidHandle 

01 (Inline Function) --------`-------- rpcrt4!CheckHandleValidity+0x9

02 (Inline Function) --------`-------- rpcrt4!EnterBufApiFull+0x94 

03 (Inline Function) --------`-------- rpcrt4!I_RpcNegotiateTransferSyntax+0x94 

04 (Inline Function) --------`-------- rpcrt4!Ndr64ClientNegotiateTransferSyntax+0xf0

05 (Inline Function) --------`-------- rpcrt4!Ndr64pClientSetupTransferSyntax+0x43f 

06 00000076`166fe6b0 00007ffe`f12d5a51 rpcrt4!NdrpClientCall3(void * pThis = 0x00000000`00000000, struct _MIDL_STUBLESS_PROXY_INFO * pProxyInfo = 0x00007ffe`f2ce32a8, unsigned long nProcNum = 8, void * pReturnValue = 0x00000076`166fe780, struct _NDR_PROC_CONTEXT * pContext = 0x00000076`166fea90, unsigned char * StartofStack = 0x00000076`166fee08 "")+0x574 

07 00000076`166fea60 00007ffe`f2c96885 rpcrt4!NdrClientCall3(struct _MIDL_STUBLESS_PROXY_INFO * pProxyInfo = 0x00007ffe`f2ce32a8, unsigned long nProcNum = 8, void * pReturnValue = 0x00000000`00000000)+0xf1 

08 (Inline Function) --------`-------- advapi32!ElfrRegisterEventSourceW+0x3a 

09 00000076`166fedf0 00007ffe`f2c96776 advapi32!ElfRegisterEventSourceW(struct _UNICODE_STRING * UNCServerName = 0x00000076`166feee0 "", struct _UNICODE_STRING * ModuleName = 0x00000076`166feed0 "Application Request Routing", void ** LogHandle = 0x00000076`166fef10)+0xd5 

0a 00000076`166feeb0 00007ffe`cdf182fc advapi32!RegisterEventSourceW(wchar_t * UNCServerName = 0x00000000`00000000 "", wchar_t * ModuleName = <Value unavailable error>)+0x36 

0b 00000076`166fef00 00007ffe`cdf1f1d7 requestrouter!FORWARDING_HANDLER::StaticInitialize(void)+0x2bc 

0c 00000076`166fef80 00007ffe`d4d18edf requestrouter!RegisterModule(unsigned long dwServerVersion = <Value unavailable error>, class IHttpModuleRegistrationInfo * pModuleInfo = <Value unavailable error>, class IHttpServer * pGlobalInfo = <Value unavailable error>)+0x4e7 

0d (Inline Function) --------`-------- iiscore!MODULE_DLL::RegisterModule+0x19 

0e (Inline Function) --------`-------- iiscore!VIRTUAL_MODULE::RegisterModule+0x2d 

0f 00000076`166ff000 00007ffe`d4d188cf iiscore!W3_SERVER::LoadModulesFromConfig(class MULTISZ * pmszModuleConfig = 0x00000228`fdb734f8, int fActuallyLoadModules = 0n1)+0x393 

10 00000076`166ff120 00007ffe`d4d18242 iiscore!W3_SERVER::InitializeGlobalModules(void)+0x3b 

11 00000076`166ff160 00007ffe`d4d3119f iiscore!W3_SERVER::Initialize(class IWorkerProcessFramework * pWpFramework = <Value unavailable error>)+0xb36 

12 00000076`166ff3d0 00007ffe`d4d3173c iiscore!IISCORE_PROTOCOL_MANAGER::InitializeGlobals(void)+0x24f 

13 00000076`166ff670 00007ffe`deb1ab07 iiscore!IISCORE_PROTOCOL_MANAGER::StartListenerChannel(wchar_t * pProtocolId = <Value unavailable error>, class IWpfListenerChannelCallback * pListenerChannelCallback = 0x00000228`fdb82830)+0x5c 

14 00000076`166ff6a0 00007ffe`deb1b9a2 w3wphost!LISTENER_CHANNEL::HandleStartListenerChannel(void)+0xdb 

15 00000076`166ff6f0 00007ffe`deb146a6 w3wphost!LISTENER_CHANNEL_START_WORKITEM::ExecuteWorkItem(void)+0x12 

16 00000076`166ff720 00007ffe`f3cdfb55 w3wphost!W3WP_HOST::ExecuteWorkItem(void * pvContext = 0x00000228`fd6b3410)+0x16 

 

advapi32!RegisterEventSourceW then set LastError to "LastErrorValue: (Win32) 0x5 (5) - Access is denied.

LastStatusValue: (NTSTATUS) 0xc0000022 - {Access Denied}  A process has requested access to an object, but has not been granted those access rights."

 

This is access denied error but a very unusual one – it does not show anything at file level or registry level access denied in ProcMon logs, and it does not give any clue when the access is denied in the time travel dump either.

 

The Code path above actually is trying to get a handle of the Application Event log, which is controlled by Application Event log ChannelAccess.

 

Application Event log ChannelAccess can be checked by Windows Command "wevtutil gl Application" in elevated Command Window.

 

The ChannelAccess right output will be something like string below:

O:BAG:SYD:(D;; 0xf0007 ;;;AN)(D;; 0xf0007 ;;;BG)(A;; 0xf0007 ;;;SY)(A;; 0x5 ;;;BA)(A;; 0x7 ;;;SO)(A;; 0x3 ;;;IU)(A;; 0x2 ;;;BA)(A;; 0x2 ;;;LS)(A;; 0x2 ;;;NS)

 

And the ChannelAccess can be written by registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog\Application\customsd.

You can manually set this registry key or set it through GPOs.

A further detailed document you can reference to: https://learn.microsoft.com/en-us/troubleshoot/windows-server/group-policy/set-event-log-security-lo...

 

Windows out of box does not set the customsd registry key and the Application Event Log ChannelAccess has a default value. If you do set the customsd registry key, then the Application Event Log ChannelAccess will use the value in the customsd registry.

 

In the issue above, the customd is set via that domain GPO so all servers in that domain cannot grab a handle of Application Event Log in the ARR RequestRouter.dll registering global module code, thus it crashes the w3wp process when it hit the code path above.

 

When setting the customsd, you need to consider any App code will use the Event Log and check ChannelAcess and make sure certain rights are assigned to allow Apps work to access Event Log in code.

Co-Authors
Version history
Last update:
‎Apr 17 2024 04:54 PM
Updated by: