Exchange 2016 CU21 to C23 Upgrade failed with Mailbox role: Client Access service -DAG Environment

Hi Team ,

We are updating the exchange CU21 to CU 23 in a DAG two server environment .The OS is windows 2016 and both servers patched to the latest patches.

Successfully patched one of the Exchange 2016 server without any problem.

On the second server , the upgrade process failed with the following error 

The following error was generated when "$error.Clear();
# Array of module names to unlock
[System.Array]
$moduleNames = "AnonymousAuthenticationModule",
"AnonymousIdentification",
"BasicAuthenticationModule",

"CertificateMappingAuthenticationModule",
"CgiModule",
"ConfigurationValidationModule",
"CustomErrorModule",

"DefaultAuthentication",
"DefaultDocumentModule",
"DigestAuthenticationModule",
"DirectoryListingModule",

"DynamicCompressionModule",
"FailedRequestsTracingModule",
"FastCgiModule",
"FileAuthorization",

"FormsAuthentication",
"HttpCacheModule",
"HttpLoggingModule",
"HttpRedirectionModule",
"IsapiFilterModule",

"IsapiModule",
"OutputCache",
"Profile",
"ProtocolSupportModule",
"RequestFilteringModule",
"RoleManager",

"ScriptModule-4.0",
"ServerSideIncludeModule",
"ServiceModel",
"ServiceModel-4.0",
"Session",

"StaticCompressionModule",
"StaticFileModule",
"UrlAuthorization",
"UrlMappingsModule",
"UrlRoutingModule-4.0",

"WindowsAuthentication",
"WindowsAuthenticationModule";
$windir = $env:windir;

$a =
&"$windir\system32\inetsrv\appcmd.exe" "set" "config" "/section:system.webServer/modules" "/lockItem:false"

Write-ExchangeSetupLog -Info ($a)

foreach($moduleName in $moduleNames)
{
$a =
&"$windir\system32\inetsrv\appcmd.exe" "set" "config" "/section:system.webServer/modules"
"/.[name=`'$moduleName`'].lockItem:false"
Write-ExchangeSetupLog -Info ($a)
}
" was run:
"System.Management.Automation.ParameterBindingException: Cannot convert 'System.Object[]' to the type 'System.String'
required by parameter 'Message'. Specified method is not supported. ---> System.NotSupportedException: Specified method
is not supported.
at System.Management.Automation.ParameterBinderBase.CoerceTypeAsNeeded(CommandParameterInternal
argument, String parameterName, Type toType, ParameterCollectionTypeInformation collectionTypeInfo, Object
currentValue)
--- End of inner exception stack trace ---
at
System.Management.Automation.CmdletParameterBinderController.VerifyArgumentsProcessed(ParameterBindingException
originalBindingException)
at
System.Management.Automation.CmdletParameterBinderController.BindCommandLineParametersNoValidation(Collection`1
arguments)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParameters(Collection`1
arguments)
at System.Management.Automation.CommandProcessor.BindCommandLineParameters()
at
System.Management.Automation.CommandProcessorBase.DoPrepare(IDictionary psDefaultParameterValues)
at
System.Management.Automation.Internal.PipelineProcessor.Start(Boolean incomingStream)
at
System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
--- End of stack
trace from previous location where exception was thrown ---
at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at
System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
at
System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][]
pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext
funcContext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at
System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)".

it seems that the web server modules on this Servers is locked.

I then tried using the  "set up assistant powershell script 

PS C:\Users\administrator.Domain\Downloads> .\SetupAssist.ps1
Setup Assist Version 23.02.14.1742

TestName Result Details
-------- ------ -------
User Administrator Passed Domain\administrator S-1-5-21-2481503423-2311206172-3694976092-500
Exchange AD Level Passed Exchange 2016 CU23
Execution Policy Warning RemoteSigned
Exchange Services Warning
Services Cache Files Failed
Computers Container Exists Passed DC=Domain,DC=DOLES
DC DNS Host Name Passed
Multiple Active Sync VDirs Detected Passed
Install Watermark Passed
Msi Cache File Passed
IIS URL Rewrite Passed Installed Version 7.2.1993
UCMA Installed Passed Installed Version: 5.0.8308.0
Microsoft Visual C++ 2012 Passed Visual C++ 2012 Redistributable
Microsoft Visual C++ 2013 Passed Visual C++ 2013 Redistributable
Other Well Known Objects Passed
Pending Reboot Passed
Valid Home MDB Passed

-----Results That Didn't Pass-----

TestName Details ReferenceInfo
-------- ------- -------------
Execution Policy RemoteSigned

Exchange Services Service: FMS Status: Stopped StartType: Set the service to Automatic and start it
Automatic
Service: MSExchangeAntispamUpdate Status:
Stopped StartType: Automatic
Service: MSExchangeDagMgmt Status: Stopped
StartType: Automatic
Service: MSExchangeDelivery Status: Stopped
StartType: Automatic
Service: MSExchangeDiagnostics Status:
Stopped StartType: Automatic
Service: MSExchangeEdgeSync Status: Stopped
StartType: Automatic
Service: MSExchangeFastSearch Status:
Stopped StartType: Automatic
Service: MSExchangeFrontEndTransport Status:
Stopped StartType: Automatic
Service: MSExchangeHM Status: Stopped
StartType: Automatic
Service: MSExchangeIS Status: Stopped
StartType: Automatic
Service: MSExchangeMailboxAssistants Status:
Stopped StartType: Automatic
Service: MSExchangeMailboxReplication
Status: Stopped StartType: Automatic
Service: MSExchangeRepl Status: Stopped
StartType: Automatic
Service: MSExchangeRPC Status: Stopped
StartType: Automatic
Service: MSExchangeSubmission Status:
Stopped StartType: Automatic
Service: MSExchangeThrottling Status:
Stopped StartType: Automatic
Service: MSExchangeTransport Status: Stopped
StartType: Automatic
Service: MSExchangeTransportLogSearch
Status: Stopped StartType: Automatic

Services Cache Files C:\ExchangeSetupLogs\ServiceStartupMode15.01. Delete the file and start Exchange Services
2507.006.xml
C:\ExchangeSetupLogs\ServiceState15.01.2507.0
06.xml

Pending Reboot HKLM:\SYSTEM\CurrentControlSet\Control\Sessio https://aka.ms/SA-RebootPending
n Manager\PendingFileRenameOperations

Setup Log Reviewer Results
--------------------------

Setup.exe Run Date: 02/18/2023 07:42:43
Setup.exe Build Number: 15.1.2507.6
Current Exchange Build: 15.1.2308.8
WARNING: Failed to determine known cause, but here is your error context that we are seeing
WARNING: Found Error:
[02/18/2023 12:23:53.0256] [2] Beginning processing Write-ExchangeSetupLog
[02/18/2023 12:23:53.0268] [2] Applied configuration changes to section "system.webServer/modules" for "MACHINE/WEBROOT/
APPHOST" at configuration commit path "MACHINE/WEBROOT/APPHOST"
[02/18/2023 12:23:53.0270] [2] Ending processing Write-ExchangeSetupLog
[02/18/2023 12:23:56.0149] [2] Beginning processing Write-ExchangeSetupLog
[02/18/2023 12:23:56.0155] [2] Applied configuration changes to section "system.webServer/modules" for "MACHINE/WEBROOT/
APPHOST" at configuration commit path "MACHINE/WEBROOT/APPHOST"
[02/18/2023 12:23:56.0156] [2] Ending processing Write-ExchangeSetupLog
[02/18/2023 12:23:58.0080] [2] Beginning processing Write-ExchangeSetupLog
[02/18/2023 12:23:58.0087] [2] Applied configuration changes to section "system.webServer/modules" for "MACHINE/WEBROOT/
APPHOST" at configuration commit path "MACHINE/WEBROOT/APPHOST"
[02/18/2023 12:23:58.0089] [2] Ending processing Write-ExchangeSetupLog
[02/18/2023 12:23:59.0322] [2] Beginning processing Write-ExchangeSetupLog
[02/18/2023 12:23:59.0329] [2] Applied configuration changes to section "system.webServer/modules" for "MACHINE/WEBROOT/
APPHOST" at configuration commit path "MACHINE/WEBROOT/APPHOST"
[02/18/2023 12:23:59.0331] [2] Ending processing Write-ExchangeSetupLog
[02/18/2023 12:23:59.0333] [1] The following 7 error(s) occurred during task execution:
[02/18/2023 12:23:59.0333] [1] 0. ErrorRecord: Cannot convert 'System.Object[]' to the type 'System.String' required by
parameter 'Message'. Specified method is not supported.
[02/18/2023 12:23:59.0334] [1] 0. ErrorRecord: System.Management.Automation.ParameterBindingException: Cannot convert '
System.Object[]' to the type 'System.String' required by parameter 'Message'. Specified method is not supported. ---> Sy
stem.NotSupportedException: Specified method is not supported.
at System.Management.Automation.ParameterBinderBase.CoerceTypeAsNeeded(CommandParameterInternal argument, String para
meterName, Type toType, ParameterCollectionTypeInformation collectionTypeInfo, Object currentValue)
--- End of inner exception stack trace ---
at System.Management.Automation.CmdletParameterBinderController.VerifyArgumentsProcessed(ParameterBindingException or
iginalBindingException)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParametersNoValidation(Collection`1 ar
guments)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParameters(Collection`1 arguments)
at System.Management.Automation.CommandProcessor.BindCommandLineParameters()
at System.Management.Automation.CommandProcessorBase.DoPrepare(IDictionary psDefaultParameterValues)
at System.Management.Automation.Internal.PipelineProcessor.Start(Boolean incomingStream)
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInterna
l[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcCo
ntext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
[02/18/2023 12:23:59.0390] [1] [ERROR] The following error was generated when "$error.Clear();
# Array of module names to unlock
[System.Array] $moduleNames = "AnonymousAuthenticationModule",
"AnonymousIdentification",
"BasicAuthenticationModule",
"CertificateMappingAuthenticationModule",
"CgiModule",
"ConfigurationValidationModule",
"CustomErrorModule",
"DefaultAuthentication",
"DefaultDocumentModule",
"DigestAuthenticationModule",
"DirectoryListingModule",
"DynamicCompressionModule",
"FailedRequestsTracingModule",
"FastCgiModule",
"FileAuthorization",
"FormsAuthentication",
"HttpCacheModule",
"HttpLoggingModule",
"HttpRedirectionModule",
"IsapiFilterModule",
"IsapiModule",
"OutputCache",
"Profile",
"ProtocolSupportModule",
"RequestFilteringModule",
"RoleManager",
"ScriptModule-4.0",
"ServerSideIncludeModule",
"ServiceModel",
"ServiceModel-4.0",
"Session",
"StaticCompressionModule",
"StaticFileModule",
"UrlAuthorization",
"UrlMappingsModule",
"UrlRoutingModule-4.0",
"WindowsAuthentication",
"WindowsAuthenticationModule";
$windir = $env:windir;
$a = &"$windir\system32\inetsrv\appcmd.exe" "set" "config" "/section:system.webServer/modules" "/lockItem:fals
e"
Write-ExchangeSetupLog -Info ($a)
foreach($moduleName in $moduleNames)
{
$a = &"$windir\system32\inetsrv\appcmd.exe" "set" "config" "/section:system.webServer/modules" "/.[name=`'
$moduleName`'].lockItem:false"
Write-ExchangeSetupLog -Info ($a)
}
" was run: "System.Management.Automation.ParameterBindingException: Cannot convert 'System.Object[]' to the type
'System.String' required by parameter 'Message'. Specified method is not supported. ---> System.NotSupportedException:
Specified method is not supported.
at System.Management.Automation.ParameterBinderBase.CoerceTypeAsNeeded(CommandParameterInternal argument, String para
meterName, Type toType, ParameterCollectionTypeInformation collectionTypeInfo, Object currentValue)
--- End of inner exception stack trace ---
at System.Management.Automation.CmdletParameterBinderController.VerifyArgumentsProcessed(ParameterBindingException or
iginalBindingException)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParametersNoValidation(Collection`1 ar
guments)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParameters(Collection`1 arguments)
at System.Management.Automation.CommandProcessor.BindCommandLineParameters()
at System.Management.Automation.CommandProcessorBase.DoPrepare(IDictionary psDefaultParameterValues)
at System.Management.Automation.Internal.PipelineProcessor.Start(Boolean incomingStream)
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInterna
l[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcCo
ntext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)".
[02/18/2023 12:23:59.0390] [1] [ERROR] Cannot convert 'System.Object[]' to the type 'System.String' required by paramete
r 'Message'. Specified method is not supported.
[02/18/2023 12:23:59.0390] [1] [ERROR] Specified method is not supported.
[02/18/2023 12:23:59.0391] [1] 1. ErrorRecord: Cannot convert 'System.Object[]' to the type 'System.String' required by
parameter 'Message'. Specified method is not supported.
[02/18/2023 12:23:59.0391] [1] 1. ErrorRecord: System.Management.Automation.ParameterBindingException: Cannot convert '
System.Object[]' to the type 'System.String' required by parameter 'Message'. Specified method is not supported. ---> Sy
stem.NotSupportedException: Specified method is not supported.
at System.Management.Automation.ParameterBinderBase.CoerceTypeAsNeeded(CommandParameterInternal argument, String para
meterName, Type toType, ParameterCollectionTypeInformation collectionTypeInfo, Object currentValue)
--- End of inner exception stack trace ---
at System.Management.Automation.CmdletParameterBinderController.VerifyArgumentsProcessed(ParameterBindingException or
iginalBindingException)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParametersNoValidation(Collection`1 ar
guments)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParameters(Collection`1 arguments)
at System.Management.Automation.CommandProcessor.BindCommandLineParameters()
at System.Management.Automation.CommandProcessorBase.DoPrepare(IDictionary psDefaultParameterValues)
at System.Management.Automation.Internal.PipelineProcessor.Start(Boolean incomingStream)
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInterna
l[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcCo
ntext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
[02/18/2023 12:23:59.0392] [1] [ERROR] The following error was generated when "$error.Clear();
# Array of module names to unlock
[System.Array] $moduleNames = "AnonymousAuthenticationModule",
"AnonymousIdentification",
"BasicAuthenticationModule",
"CertificateMappingAuthenticationModule",
"CgiModule",
"ConfigurationValidationModule",
"CustomErrorModule",
"DefaultAuthentication",
"DefaultDocumentModule",
"DigestAuthenticationModule",
"DirectoryListingModule",
"DynamicCompressionModule",
"FailedRequestsTracingModule",
"FastCgiModule",
"FileAuthorization",
"FormsAuthentication",
"HttpCacheModule",
"HttpLoggingModule",
"HttpRedirectionModule",
"IsapiFilterModule",
"IsapiModule",
"OutputCache",
"Profile",
"ProtocolSupportModule",
"RequestFilteringModule",
"RoleManager",
"ScriptModule-4.0",
"ServerSideIncludeModule",
"ServiceModel",
"ServiceModel-4.0",
"Session",
"StaticCompressionModule",
"StaticFileModule",
"UrlAuthorization",
"UrlMappingsModule",
"UrlRoutingModule-4.0",
"WindowsAuthentication",
"WindowsAuthenticationModule";
$windir = $env:windir;
$a = &"$windir\system32\inetsrv\appcmd.exe" "set" "config" "/section:system.webServer/modules" "/lockItem:fals
e"
Write-ExchangeSetupLog -Info ($a)
foreach($moduleName in $moduleNames)
{
$a = &"$windir\system32\inetsrv\appcmd.exe" "set" "config" "/section:system.webServer/modules" "/.[name=`'
$moduleName`'].lockItem:false"
Write-ExchangeSetupLog -Info ($a)
}
" was run: "System.Management.Automation.ParameterBindingException: Cannot convert 'System.Object[]' to the type
'System.String' required by parameter 'Message'. Specified method is not supported. ---> System.NotSupportedException:
Specified method is not supported.
at System.Management.Automation.ParameterBinderBase.CoerceTypeAsNeeded(CommandParameterInternal argument, String para
meterName, Type toType, ParameterCollectionTypeInformation collectionTypeInfo, Object currentValue)
--- End of inner exception stack trace ---
at System.Management.Automation.CmdletParameterBinderController.VerifyArgumentsProcessed(ParameterBindingException or
iginalBindingException)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParametersNoValidation(Collection`1 ar
guments)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParameters(Collection`1 arguments)
at System.Management.Automation.CommandProcessor.BindCommandLineParameters()
at System.Management.Automation.CommandProcessorBase.DoPrepare(IDictionary psDefaultParameterValues)
at System.Management.Automation.Internal.PipelineProcessor.Start(Boolean incomingStream)
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInterna
l[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcCo
ntext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)".
[02/18/2023 12:23:59.0392] [1] [ERROR] Cannot convert 'System.Object[]' to the type 'System.String' required by paramete
r 'Message'. Specified method is not supported.
[02/18/2023 12:23:59.0392] [1] [ERROR] Specified method is not supported.
[02/18/2023 12:23:59.0392] [1] 2. ErrorRecord: Cannot convert 'System.Object[]' to the type 'System.String' required by
parameter 'Message'. Specified method is not supported.
[02/18/2023 12:23:59.0393] [1] 2. ErrorRecord: System.Management.Automation.ParameterBindingException: Cannot convert '
System.Object[]' to the type 'System.String' required by parameter 'Message'. Specified method is not supported. ---> Sy
stem.NotSupportedException: Specified method is not supported.
at System.Management.Automation.ParameterBinderBase.CoerceTypeAsNeeded(CommandParameterInternal argument, String para
meterName, Type toType, ParameterCollectionTypeInformation collectionTypeInfo, Object currentValue)
--- End of inner exception stack trace ---
at System.Management.Automation.CmdletParameterBinderController.VerifyArgumentsProcessed(ParameterBindingException or
iginalBindingException)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParametersNoValidation(Collection`1 ar
guments)
at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParameters(Collection`1 arguments)
at System.Management.Automation.CommandProcessor.BindCommandLineParameters()
at System.Management.Automation.CommandProcessorBase.DoPrepare(IDictionary psDefaultParameterValues)
at System.Management.Automation.Internal.PipelineProcessor.Start(Boolean incomingStream)
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInterna
l[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcCo
ntext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
[02/18/2023 12:23:59.0393] [1] [ERROR] The following error was generated when "$error.Clear();
# Array of module names to unlock
[System.Array] $moduleNames = "AnonymousAuthenticationModule",
"AnonymousIdentification",
"BasicAuthenticationModule",
"CertificateMappingAuthenticationModule",
"CgiModule",
"ConfigurationValidationModule",
"CustomErrorModule",
"DefaultAuthentication",
"DefaultDocumentModule",
"DigestAuthenticationModule",
"DirectoryListingModule",
"DynamicCompressionModule",
"FailedRequestsTracingModule",
"FastCgiModule",
"FileAuthorization",
"FormsAuthentication",
"HttpCacheModule",
"HttpLoggingModule",
"HttpRedirectionModule",
"IsapiFilterModule",
"IsapiModule",
"OutputCache",
"Profile",
"ProtocolSupportModule",
"RequestFilteringModule",
"RoleManager",
"ScriptModule-4.0",
"ServerSideIncludeModule",
"ServiceModel",
Looks like we weren't able to determine the cause of the issue with Setup.
Address all Failed tests above, if any.
If that doesn't find the cause, please notify 'email address removed for privacy reasons' to help us improve the scripts.

Please  help 

Can I run the setup again or should I reboot the server and try the upgrade again ?

or should I go with the recover option ?

Regards

Rk