Released: August 2023 Exchange Server Security Updates

We have Exchange Server 2016 CU23 and are solely using this in a management perspective for Exchange Online mailboxes in hybrid. This is an English language server.

We had the update installed via Azure Update Management Center. Within UMC the update had a status of failed. I jumped on the server and all Exchange services were running and I ran https://microsoft.github.io/CSS-Exchange/Diagnostics/HealthChecker/, which reported no issues and the correct build number for this update. I then ran https://microsoft.github.io/CSS-Exchange/Security/CVE-2023-21709/, as per the https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2023-21709. At this point all seemed to be well and I browsed to ECP to ensure it was up, I didn't login however, all good I thought.

That was Wednesday 9th. The next day a report comes in that another member of IT can't login to ECP. The login page is up but upon login it throws an IIS error. The same as this one https://learn.microsoft.com/en-us/exchange/troubleshoot/client-connectivity/owa-stops-working-after-update. - Server Error in '/ecp' Application. Could not load file or assembly 'Microsoft.Exchange.Common - I followed the steps in the article to run CMD as administrator and then call the Security Update installer. I noted that the article mentions the MSP file but all that was available was the EXE, so this is what I went with. The GUI launched and the install started. After a few minutes I received a message about how a previous install by the user SYSTEM had failed and it needed to be rolled back, I selected OK and it then prompted a reboot. The reboot took over an hour and when it came back all services were set to Disabled. We manually set them back to Automatic but none of them will start.

This prompted further investigation and it was at this point we found this article, among others, detailing issues with this update. We've attempted to run ServiceControl.ps1 AfterPatch. This failed stating the cmdlet Start-SetupService wasn't an available cmdlet. After further reading and investigation we created an alias for Start-SetupService to Start-Service instead. This now results in the script failing with the following:

PS C:\Program Files\Microsoft\Exchange Server\V15\Bin> .\ServiceControl.ps1 AfterPatch
Start-Service : A parameter cannot be found that matches parameter name 'IgnoreTimeout'.
At C:\Program Files\Microsoft\Exchange Server\V15\Bin\ServiceControl.ps1:561 char:79
+ ... e $serviceName -ev script:serviceControlError -IgnoreTimeout:$IgnoreT ...
+ ~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Start-Service], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.Commands.StartServiceCommand

We're now at a point where none of the Exchange services will start.

Control Panel shows KB5025903 as the installed version of Exchange, which is the previous SU.

I've tried all manner of setup.exe for CU23 using the /m:install, upgrade and repairserver options. None of them succeed.

I don't know what to try next. Any suggestions as to how we recover this would be VERY welcome! We're 24 hours in now.

Just to add, we have valid backups of the Azure VM that runs Exchange in both Veeam Agent and native Azure backups and snapshots from points in time prior to the installation of the Aug SU. Is it safe to simply shutdown the current broken server and restore from one of these backups? Historically you have to be careful with restoring Exchange, but this is a single server with no local mailboxes and is purely used to management purposes.