Resolving Exchange Server IIS 500 - Maintenance Error
Hi All, This is my first post here, ment as advice / troubleshoot tips for Exchange admins. I see regularly Exchange admins having trouble with IIS 500 page errors. This topic is not rather a question, but some information I would like to share with admins that run into an IIS500 Error (reason: Maintenance) on Exchange OWA and ECP after installing .NET Updates. The page of ECP (and OWA) looks like the follow page below. Please note that IIS 500 can also be triggered by other errors, this is for specific the situation written below. If you have recreated the virtual directories, you find out that the problem is still there. The example here is based on Exchange 2019 / Windows Server 2019, but I have also seen this behavior on Exchange Server 2016 and Exchange Server 2013 on Windows Server 2016 and 2012R2. Rule one: Give Exchange servers some time booting up, don't try to immediately logon to ECP/OWA when the logon screen beach appears on the console. Check the CPU up-time, and I recommend waiting at least wait for 5 to 10 minutes uptime (CPU time) and verify the server CPU usage has normalized. IIS 500 common error checks Although IIS 500 is quite a common error, there are some occasions where you should figure out if you have installed Windows / .NET Updates on the server you are working on. Please examine the server eventlog for details in the “Windows Logs > Setup” section. If so? You should first verify that all Exchange Services are running correctly that have the startup type to start automatically. In environments we manager we regularly see the following services not starting after a reboot of the server: MS Exchange Replication (note: also for single server setups) MS Exchange Transport Log Search. MS Exchange Compliance Audit. Some others, but most are running OK. Note: When rebooting your system, the services not starting can vary. You will also find: Eventlog Application Error Exchange Compliance Audit because “Mailbox database is not available”. Eventlog Application Error ActiveSync cannot open mailboxes. Other Exchange errors referring that it is unable to do some “things”. Server memory usage is lower than normal after a reboot. What should you do? First, start the Exchange Services that aren’t running and have startup type set to automatically. Start with back-end services like MS Exchange Replication and give it some time and lastly start services that are for log searching or auditing. When observing your performance monitor you will probably see a surge in resource usage. In the Eventlog > Application you will see some Exchange things to come to life again. After a few minutes, when you login to ECP you will see the environment again (note: the first time someone logs in to a virtual directory the web-pool needs to start, this might take a few seconds). Restarted the server and it’s broken again! If you restart the server again, you will probably run into the same situation again. Why? The root cause is not known to me, but it has to do something with .Net Framework updates. What you need to do is basically waiting. So, don’t reboot the server if you have your pages working again by starting the services, etc. Rebooted the server? You can do the actions written above again. .Net Optimization Proces For it to permanently work after reboots you have to wait for a process to start working on the .Net optimization. When the server is “idle”, usually 20-30 minutes after booting up the process “.Net Optimization Task” starts running on your Exchange server. This usually takes up to 20-40 minutes. When it’s done you will see most of the time (not always) notifications in eventlog: Application > .NET Runtime Optimization Service > EventID 1130. .NET Runtime Optimization Service (4.0.30319.0) - Installed from repository: mscorlib When the task is completed and you restart your server again, it should be running fine again. You don’t have to reboot the server If you have started the services manually. The key is they don’t want to start automatically unless the “.Net Optimization Task” has done it’s thing. What actually happens, I don’t know, maybe some of the Exchange Team members can give an answer, but this is what I’ve found out in many environments (test production, etc). It doesn’t matter if you install updates with Antimalware software switched on or off. I hope this will help some users having trouble and starting to fix thing’s that actually aren’t broken, but just need some special attention.
Exchange System Mailboxes after migration to Exchange 365
Hello all, After the migration off the mailboxes to Exchange 365, there are 2 mailboxes in our on-premise environment: extest + DiscoverySearchMailbox. Because of this 2 mailboxes, the Exchange server has the 'Mailbox Server'-role. I think this is not neccesarry. I don't understand why this 2 mailboxes are counted as normal mailbox, because other system mailboxes aren't and also not visible. Does anyone know what the function is off this mailboxes? Kind regards, Arjan
Exchange 2019 error 2004 cannot log in to owa with any new user?
Hello, I am faced with a very bizarre situation which I have never encountered before perhaps someone has seen this. I have an on-prem exchange 2019 server. Everything works fine, all users can connect but I cannot create any new users. I explain, I create a user in AD, assign it a mailbox, everything performs correctly. But when I try to add the mailbox to outlook or try to log in via OWA I always get incorrect password. No matter what I do. One thing I noticed is that in my ECP, prior to making this new user I had 3 self signed certificates that were invalid. I simply clicked "renew" and they all became valid. However, now in my event viewer I keep getting the following warning Unable to find the certificate with thumbprint in the current computer or the certificate is missing private key. The certificate is needed to sign the outgoing token. A little googling and it seems this error is related to Microsoft Exchange Server Auth Certificate but mine is valid. However, I'm starting to think that I cannot connect any mailboxes because somewhere in IIS it is not finding the correct certificate, or maybe it is not bound properly? I saw a similar post which says if we already have a valid certificate we can run the following command Set-AuthConfig -NewCertificateThumbprint THISONEEXISTSINEXCHANGE -NewCertificateEffectiveDate (Get-Date) but I'm unsure what this will do or if it will fix the issue. Before I do anything I'd like to get some expert opinions on this, have anyone ever come across this kind of issue? Thanks
Federation Relationship issue
Hi, We have a classic Hybrid configuration with several on-premises Exchange 2019 CU12 servers. Everything works as expected but we fail with Test-FederationRelationship cmdlet. On-premises servers: Get-OrganizationRelationship | Test-OrganizationRelationship -UserIdentity <my email> Begin testing for organization relationship CN=On-premises to O365 - <some GUID>,CN=Federation,CN=<our organization>,CN=Microsoft Exchange,CN=Services,CN=Configuration,<our domain>, enabled state True. Exchange D-Auth Federation Authentication STS Client Identities are urn:federation:MicrosoftOnline/FYDIBOHF25SPDLT.<our domain>; WARNING: An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object. Object reference not set to an instance of an object. + CategoryInfo : NotSpecified: (:) [Test-OrganizationRelationship], NullReferenceException + FullyQualifiedErrorId : System.NullReferenceException,Microsoft.Exchange.Management.Sharing.TestOrganizationRelationship + PSComputerName : <any server> When I test the trust, it returns ok: Test-FederationTrust -UserIdentity <my email> Begin process. STEP 1 of 6: Getting ADUser information for <my email>... RESULT: Success. STEP 2 of 6: Getting FederationTrust object for <my email>... RESULT: Success. STEP 3 of 6: Validating that the FederationTrust has the same STS certificates as the actual certificates published by the STS in the federation metadata. RESULT: Success. STEP 4 of 6: Getting STS and Organization certificates from the federation trust object... RESULT: Success. Validating current configuration for FYDIBOHF25SPDLT.<our domain>... Validation successful. STEP 5 of 6: Requesting delegation token... RESULT: Success. Token retrieved. STEP 6 of 6: Validating delegation token... RESULT: Success. Closing Test-FederationTrust... RunspaceId : 5cbacaf9-78ab-45a3-ab89-029cb4ced097 Id : FederationTrustConfiguration Type : Success Message : FederationTrust object in ActiveDirectory is valid. RunspaceId : 5cbacaf9-78ab-45a3-ab89-029cb4ced097 Id : FederationMetadata Type : Success Message : The federation trust contains the same certificates published by the security token service in its federation metadata. RunspaceId : 5cbacaf9-78ab-45a3-ab89-029cb4ced097 Id : StsCertificate Type : Success Message : Valid certificate referenced by property TokenIssuerCertificate in the FederationTrust object. RunspaceId : 5cbacaf9-78ab-45a3-ab89-029cb4ced097 Id : StsPreviousCertificate Type : Success Message : Valid certificate referenced by property TokenIssuerPrevCertificate in the FederationTrust object. RunspaceId : 5cbacaf9-78ab-45a3-ab89-029cb4ced097 Id : OrganizationCertificate Type : Success Message : Valid certificate referenced by property OrgPrivCertificate in the FederationTrust object. RunspaceId : 5cbacaf9-78ab-45a3-ab89-029cb4ced097 Id : TokenRequest Type : Success Message : Request for delegation token succeeded. RunspaceId : 5cbacaf9-78ab-45a3-ab89-029cb4ced097 Id : TokenValidation Type : Success Message : Requested delegation token is valid. On cloud: (Get-OrganizationRelationship)[1] | Test-OrganizationRelationship -UserIdentity <my email> Begin testing for organization relationship CN=O365 to On-premises - <some GUID>,CN=Federation,CN=Configuration,CN=<our organization>.onmicrosoft.com,CN=ConfigurationUnits,DC=EURPR04A007,DC=PROD,DC=OUTLOOK,DC=COM, enabled state True. Exchange D-Auth Federation Authentication STS Client Identities are uri:WindowsLiveID/outlook.com;urn:federation:MicrosoftOnline/outlook.com; STEP 1: Validating user configuration RESULT: Success. STEP 2: Getting federation information from remote organization... RESULT: Unable to retrieve federation information from remote organization. Doing local testing only. STEP 3: Requesting delegation token from the STS... RESULT: Success. Retrieved token for target https://<our access point>/autodiscover/autodiscover.svc/wssecurtiy for offer Name=MSExchange.Autodiscover,Duration=28800(secs) STEP 4: Getting organization relationship settings from remote partner... RESULT: Unable to retrieve organization relationships from remote organization. RESULT: Error. LAST STEP: Writing results... Identity : Id : AutodiscoverServiceCallFailed Status : Error Description : The Autodiscover call failed. IsValid : True ObjectState : New COMPLETE. WARNING: The federated domain <our domain> of the user is in the local organizational relationship which normally only contains the domains of external organizations. I didn't find any clues that could help in troubleshooting of the issue. Any ideas? King regards, Dmitry
Exchange 2013/2019 coexistence and client connectivity to new 2019 problem
Hey all, Some details first: Forest/Domain functional level: 2012 R2 (only one domain) 2 x Exchange 2013 CU 23 Servers in a DAG (both on Server 2012) 1 x Exchange 2019 CU 12 server (Server 2022) No load balancers or anything between clients and the Exchange Servers Outlook 365 on Windows 10 Have installed Exchange 2019 into an Exchange 2013 environment. All dns still pointing to the 2013 servers so everything is functional for users. The scp for the new 2019 server is pointing to the 2013 environment. Client connectivity to the new server doesn't appear to be working though, or the new server can't proxy requests to the server holding the mailbox. I have a test PC that I change the hosts file on to point autodiscover.domain.com and webmail.domain.com to the new server. Upon opening Outlook 365 it opens but there is no mailflow to/from the client. Outlook cannot update folders. Outlook connection status does not show any connection to the new server. From the test pc, pinging the 2 urls produces the correct IP address and if I browse to https://webmail.domain.com/ecp I get the console for the 2019 server. The 2019 server shows in the list of servers alongside the 2013 ones in the ecp so I assume coexistence is functional. I have looked through various logs but cannot see anything that would indicate why Outlook 365 is not be able to communicate with the 2019 server. No firewall issues that I can see. In the iis logs on the 2019 server I can see what I assume is my Outlook 365 client communicating as I am seeing "POST /mapi/emsmdb" with my PC's IP and the user agent "Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.16327;+Pro)" Can anyone think of what the issue may be, or where else I can look to troubleshoot? thanks jcCVE-2024-49040: Mitigating a Critical Microsoft Exchange Server Vulnerability
CVE-2024-49040 is a spoofing vulnerability identified in Microsoft Exchange Server versions 2016 and 2019. This flaw allows attackers to forge legitimate sender addresses on incoming emails, potentially making malicious messages appear trustworthy. The vulnerability arises from improper verification of the P2 FROM header during email transport, permitting non-RFC 5322 compliant headers to pass through and be displayed as legitimate by email clients like Microsoft Outlook. Recommended Mitigation Steps To protect your organization from this vulnerability, consider the following steps: Apply Security Patches: Enhance Email Security: Educate Users: Implement Strong Password Policies: Monitor Network Traffic: By taking these steps, organizations can significantly reduce the risk of exploitation and protect their sensitive data. It is essential to stay informed about the latest security threats and to adopt a proactive approach to cybersecurity. These patches are available in WSUS. If the concerned team has not yet synchronized, please proceed with the synchronization and apply the latest patches. Alternatively, you can find these patches on the official Note: These patches are applicable for the following Exchange versions: Microsoft Exchange Server 2016 Cumulative Update 23 Microsoft Exchange Server 2019 Cumulative Update 14 Microsoft Exchange Server 2019 Cumulative Update 13ECP on Exchange 2019 Server doesn't run without an Exchange 2016 Server
I'm in the same situation as I'm sure many others are, migrating from on-prem Exchange 2016 to MS 365, then wanting to replace the on-prem Exchange 2016 Server with an Exchange 2019 Server just for running ECP, with no local mailboxes. So far, I have: Migrated all the mailboxes in use to MS365 There are a fair few mailboxes that are no longer required, so haven't been migrated. These are to be retired. Installed an Exchange 2019 Server on Windows Server 2022 (Server Core) Installed Certificates on the new Exchange 2019 server for webmail Changed DNS records for Exchange HTTPS services to point to just the new 2019 Server Run the HCW on the 2019 box to re-configure Hybrid Configuration to use the new server Having done all of that, I've powered down the old Exchange 2016 server to test that everything works on just the new 2019 server. It turns out that ECP is working when both servers are online, but as soon as the 2016 server is shut down, ECP (running from the 2019 server) stops working. I can confirm that the 2019 server is serving the pages, as it's a new certificate than was used on the 2016 server. The login page loads correctly, but after entering my credentials it returns HTTP 503 "<webserver> can't currently handle this request." Once the 2016 server is brought back online it works again. Is this expected behaviour because Exchange is unable to contact one of it's servers, that would disappear once it's cleanly decommissioned, or is there something wrong with the new server? I've completely removed and re-installed Exchange on this box, but it doesn't seem to have made a difference.
