Hi everyone,
We are in the process of decommissioning our Exchange 2013 servers in a Hybrid environment with Exchange 2016 and Exchange 2013 coexistence. All servers running the latest or second latest SU/CU.
We have removed all mailboxes and databases from the Exchange 2013 servers, and as a part of the decommissioning process we usually shut down the servers for 4 weeks before we uninstall the application software in order to find any unknown integrations before the servers are deleted.
After shutting down our Exchange 2013 servers we are having issues with an application making EWS requests getting a HTTP 503 service unavailable.
These requests go from the application server to a BigIP HLB, which then distributes the traffic between two Exchange 2016 servers. But for some reason the Exchange 2016 servers then proxy's the requests to our offline Exchange 2013 hosts so we end up with the service unavailable error.
I'm trying to figure out why Exchange 2016 is proxying the requests instead of processing them directly. The service account running all the EWS commands used to have a mailbox on Exchange 2013, but has been migrated to Exchange Online and now only has a remote mailbox object in the on-prem environment.
I see some references to a "cache entry" for backend servers in the logs, but not sure if i need to somehow clear that. Or if maybe i need to set the EWS url's to $null like one would do with autodiscover when you want to remove the SCP?
Example from the IIS logs for EWS:
2023-01-16 23:59:59 10.10.10.10 POST /EWS/Exchange.asmx &CorrelationID=<empty>;&cafeReqId=e958c772-3a0d-4cff-bc7d-4b7e372b74bd; 443 domain\account x.x.x.x ExchangeServicesClient/0.0.0.0 - 503 0 0 5614
Examples from http proxy logs for EWS: (i removed/obfuscated some of the info)
2023-01-17T05:01:31.532Z,984a24e6-e2a2-479f-a9a9-c1840824ee18,15,1,2507,16,,Ews,webmail.domain.com,/EWS/Exchange.asmx,,NTLM,true,netbiosdomain\account,,Sid,ExchangeServicesClient/0.0.0.0,10.x.x.x,exchange2016server,503,ConnectFailure,ConnectFailure,POST,Proxy,exchange2013server.netbiosdomain.domain.com,15.00.1497.000,IntraForest,WindowsIdentity,Database~89b52aec-2fde-409e-9396-e8b91986f4a1~~2023-02-16T05:01:25~netbiosdomain.domain.com~1,,,1286,,,,0,0,,0,,0,,0,0,,0,5605,0,,,,,,,,2802,0,5605,0,,5605,,5605,5605,,,,BeginRequest=2023-01-17T05:01:25.925Z;CorrelationID=<empty>;ProxyState-Run=None;ServerLocatorRefresh=2fdc8a46-685e-4133-816e-3b1930dc95a2;RefreshingCacheEntry=CacheEntry(BackEndServer exchange2013server.netbiosdomain.domain.com~1941997017|ResourceForest netbiosdomain.domain.com|FailoverSequenceNumber 638079945647364933|LastRefreshTime 03.01.2023 11.36.42);RandomBE=exchange2013server.netbiosdomain.domain.com~1941997017;FEAuth=BEVersion-1941997017;NewConnection=10.134.179.22&0;BeginGetRequestStream=2023-01-17T05:01:28.729Z;OnRequestStreamReady=2023-01-17T05:01:31.531Z;InvalidatingBackEndServerCache=89b52aec-2fde-409e-9396-e8b91986f4a1;ProxyState-Complete=ProxyRequestData;SharedCacheGuard=0;EndRequest=2023-01-17T05:01:31.532Z;S:ServiceCommonMetadata.Cookie=fed0ab9fb3e1425d88f3080276f1cc1f,WebExceptionStatus=ConnectFailure;WebException=System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time or established connection failed because connected host has failed to respond x.x.x.x:444
I would appreciate any insight or tips, thanks.
