Blog Post

Exchange Team Blog

7 MIN READ

Email Stuck in Exchange On-premises Transport Queues

Microsoft

Jan 01, 2022

We have addressed the issue causing messages to be stuck in transport queues of on-premises Exchange Server 2016 and Exchange Server 2019. The problem relates to a date check failure with the change of the new year and it not a failure of the AV engine itself. This is not an issue with malware scanning or the malware engine, and it is not a security-related issue. The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues.

We have now created a solution to address the problem of messages stuck in transport queues on Exchange Server 2016 and Exchange Server 2019 because of a latent date issue in a signature file used by the malware scanning engine within Exchange Server. Customer action is required to implement this solution. When the issue occurs, you’ll see errors in the Application event log on the Exchange Server, specifically event 5300 and 1106 (FIPFS), as illustrated below:

Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 1:03:42 AM 
Event ID: 5300 
Level: Error 
Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.

Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 11:47:16 AM 
Event ID: 1106 
Level: Error 
Computer: server1.contoso.com 
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

Using the Automated Solution

Download the script here: https://aka.ms/ResetScanEngineVersion
Before running the script, change the execution policy for PowerShell scripts by running Set-ExecutionPolicy -ExecutionPolicy RemoteSigned.
Run the script on each Exchange mailbox server that downloads antimalware updates in your organization (use elevated Exchange Management Shell).

Edge Transport servers are unaffected by this issue. You can run this script on multiple servers in parallel. After the script has completed, you will see the following output:

[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Reset-ScanEngineVersion.ps1
EXCH1 Stopping services...
EXCH1 Removing Microsoft engine folder...
EXCH1 Emptying metadata folder...
EXCH1 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EXCH1 Starting engine update...
Running as EXCH1-DOM\Administrator.
--------
Connecting to EXCH1.CONTOSO.com.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------
[PS] Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
--------
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>Get-EngineUpdateInformation

Engine                : Microsoft
LastChecked           : 01/01/2022 08:58:22 PM -08:00
LastUpdated           : 01/01/2022 08:58:31 PM -08:00
EngineVersion         : 1.1.18800.4
SignatureVersion      : 1.355.1227.0
SignatureDateTime     : 01/01/2022 03:29:06 AM -08:00
UpdateVersion         : 2112330001 (note: higher version number starting with 211233... is also OK)
UpdateStatus          : UpdateAttemptSuccessful

Using the Manual Solution

In lieu of using the script, customers can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange mailbox server in your organization that downloads antimalware updates. Edge Transport servers are unaffected by this issue.

Verify the impacted version is installed
Run Get-EngineUpdateInformation and check the UpdateVersion information. If it starts with "22..." then proceed. If the installed version starts with "21..." you do not need to take action.

Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service. When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.

Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001 (or higher)

After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.

FAQ

I'm not sure if this issue affects my organization. How do I find out?
Run the latest release of the HealthChecker script (https://aka.ms/ExchangeHealthChecker) on every Exchange server in your organization and check for the FIP-FS warning which will be shown if your server is affected and further actions are required.

Is the solution for this problem automated?
Implementation of the solution requires customer actions, and it will take some time to make the necessary changes, download the updated files, and clear the transport queues. Actions can be automated with the scan engine reset script from https://aka.ms/ResetScanEngineVersion or they can be performed manually. Whether you perform the steps automatically or manually, they must be performed on every on-premises Exchange 2016 and Exchange 2019 server in your organization. If you use the automated script, you can run it on multiple servers in parallel.

How long will running of automated script take?
Depending on the size of your organization, the script might take some time to run; please be patient.

How long will it take to clear up the queues after the script has been run?
Depending on the number of messages that were queued up and the amount of new messages transport has to process, the time might vary. Please be patient and monitor those queues are draining (number of messages are decreasing) by using Get-queue command.

We are in Exchange Hybrid environment. What do we need to do?
If you are using your on-premises Exchange server to send email (for example using Centralized Mailflow or sending messages from on-premises devices), please follow this blog post and use the script to change configuration on your on-premises servers used for email transport. If you are using Exchange on-premises only for management of Exchange recipients, you do not need to take any action.

What are the services that the script is stopping?
The following services will be restarted: Microsoft Filtering Management and Microsoft Exchange Transport.

We have temporarily disabled antimalware. Should it be enabled after following this blog post?
If you have temporarily disabled the antimalware service, you should enable it after you have followed this blog post (use the Enable-AntimalwareScanning.ps1 script). The solution described in this post is a full solution for this problem and will result in transport queues clearing and antimalware engine working as expected.

The version of the updated scan engine starts with 2112330001 (or higher); is this right? Should we be concerned that it seems to reference a date that does not exist?
The newly updated scanning engine is fully supported by Microsoft. While we need to work on this sequence longer term, the scanning engine version was not rolled back, rather it was rolled forward into this new sequence. The scanning engine will continue to receive updates in this new sequence.

What if my Exchange servers do not have access to the Internet?
If your Exchange mailbox servers do not download antimalware updates from the Internet, you do not need to perform any manual action. In that case, the servers have not been downloading antimalware updates to begin with, and the problem described here will not exist.

We have an Exchange 2013 server and while there are no crashes, I see the server has the problem engine version starting with "22...". What should we do?
Exchange Server 2013 is not impacted by transport crashes so there will be no buildup of email in transport queues. If your Exchange 2013 server took the antimalware update and it is now on version starting with "22..." you should use the automated or manual steps in this blog post to get your server on an engine version "21..." to continue getting the antimalware updates. Update 1/11/2022: Exchange 2013 customers - before running the script, please check if the engine got updated automatically to version 21... additional mechanism exists that updates the engine to the latest version and by now, your Exchange 2013 servers should have auto-corrected the version.

Script generates error “WARNING: Unable to process update request because engine metadata is not available. Attempting to synchronize” metadata. Please try to run the cmdlet again later.
For the Exchange servers accessing Internet via proxy:

Launch Exchange Management Shell
Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell
Set-ProxySettings -Enabled $true -Server <ProxyServer> -Port <proxy.port>
Re-run the script

If still not resolved:

Copy msvcr110.dll from c:\windows\system32 to %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Bin and %ProgramFiles%\Microsoft\Exchange Server\V15\Bin
Restart the Exchange server
Re-run the script

I have many Exchange servers in my environment; is there a way to locally distribute the definition files?

Use the steps in this article and download update files to a server. Example: Update-Engines.ps1 -EngineDirPath C:\ScanEngineUpdates\
Share the folder on which the update files were copied, for example \\server1\amware
Run the following command on all remaining servers that need to copy the update files: Set-MalwareFilteringServer -PrimaryUpdatePath \\server1\amware -Identity mail1.contoso.com
Re-run the reset scan engine script on all the servers.

Major changes to this blog post:

1/11: Added more information to Exchange 2013 FAQ regarding engine auto-update
1/4: Added a FAQ on how to find impacted servers using Health Checker script
1/3: Added a FAQ for local distribution of updates within the organization
1/3: Added a FAQ related to Proxy configuration and script error
1/3: Added a FAQ for Exchange 2013 servers
1/3: Addition of Add-PSSnapin to automated process; various other smaller changes
1/2: Clarified the exact process to run the automated script solution; various other smaller changes and clarifications
1/2: Added the FAQ section to the blog post
1/1: Major update mentioning our manual and scripted solution for this problem; disablement of Antimalware service as a workaround has been removed
1/1: Original release

The Exchange Team

Updated Jan 11, 2022

Version 36.0

Microsoft

Joined April 19, 2019

View Profile

Exchange Team Blog

You Had Me at EHLO..

Larry Heier
Copper Contributor
Jan 03, 2022
Microsoft,

First off thank you all for coming up with quick workarounds and a solution to this unfortunate issue. It has been a long year of emergency fixes/security patches for Exchange.

This brings up the question about the retirement of Exchange hybrid servers. We've heard for a very long time of a future replacement for Exchange hybrid so we don't have to maintain an on-premise Exchange server which is required for O365 support with Azure AD Connect. We are tired of patching these hybrid servers and we see customers retiring such servers even though they move to an "unsupported" state.

When will there be another supported way to manage Exchange attributes on-premise and a substitute to more easily allow systems/devices to relay off an internal email server?

It is long overdue Microsoft and hoping someone from the Exchange team can comment on the plans to simplify our IT lives so we don't have have to upgrade/patch Exchange hybrid servers every 2 to 6 months to stay in support and stay secure.

thank you,
Larry Heier
MarkDePalma
Copper Contributor
Jan 01, 2022
I wrote a script and article on how to roll-back the engine update, so that that you don't need to disable the engine. This also resolves transport rule issues.

https://blog.markdepalma.com/?p=810
Dani Kaplan
Copper Contributor
Jan 01, 2022
Thanks, I'll try to run this. who pushes updates on 31/12 to production ?!!!!!
Frank Carius
MVP
Jan 01, 2022
Thanks for that quick status message.
Question: Exchange normally removes not delivered messages after 48h ( "MessageExpirationTimeout" https://docs.microsoft.com/en-us/exchange/mail-flow/queues/configure-message-intervals?view=exchserver-2019#configure-the-message-expiration-timeout-interval) Does this also affect messages on "Submission queues". This could really make it worse if companies loose "new year" messages before admins are back from vacation/Weekend. On the other side: if could trigger smaller companies without 24x7 operations to migrate to exchange online. 🙂

Frank
ScottSchnoll
Microsoft
Jan 01, 2022
atsolomon, if after disabling the agent and restarting transport it still doesn't work, then check for event 4010. It will list any transport rules that depend on the malware agent which is causing mail to queue after remediating the issue. Disable or adjust that rule to resolve.
JasonNash
Copper Contributor
Jan 02, 2022
Script worked fine here. Just a few observations.. make sure you run as admin, update/download signatures seems to be taking some time. Over half and hour this morning. I also had to restart the Transport service manually following the reset scan engine script. I had previously ran the disable antimalware script as a temporary workaround and when then running the enable antimalware script it sits and appears to hang for a long time, this again appears to be due to the signature update download but it will eventually complete.
GRauth
Brass Contributor
Jan 02, 2022
Ron-64Had the same issue, have a look below how I managed this.
ThomasStensitzki-MVP : This worked instead:
Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell
[PS] C:\>Start-EngineUpdate -Verbose
VERBOSE: [08:24:47.367 GMT] Start-EngineUpdate : Beginning processing
VERBOSE: [08:24:47.961 GMT] Start-EngineUpdate : Initiating engine Update für the following engines: Microsoft
VERBOSE: [08:24:48.149 GMT] Start-EngineUpdate : Ending processing

After some small other issues, mail transport works again.
g-evans I added you, since you have the same issue.
GRauth
Brass Contributor
Jan 02, 2022
Well, while this is not working; I still wonder why Microsoft's solution is a security rollback. "Just use our old engine." That is a solution. But on a very short term.

Microsoft is working around the clock to fix this! Microsoft apologizes for the inconvenience that issue caused. In 1999, we had meetings and symposiums all about a so-called Millennium Bug. The world was afraid that engineers could have used strings that can't hold a 4 digit value for the year. Now 22 years later, Microsoft has done it. Companies, where the IT Department is not on duty because New Year's Day and the following Sunday, will have a problem on Monday. Lost E-Mails. Microsoft will tell them, they should have used Exchange online instead.

This is no open-source software maintained by a community. We paid for the software licence. But most valued information you find to this issue is in the community forums instead of Microsoft official platforms.
aTorreZ
Copper Contributor
Jan 02, 2022
In my case the script failed to stop the services because of a time-out. So, I "helped" the script stopping the services manually.
Make sure you run the script as an administrator.
John_C_Kirk
Copper Contributor
Jan 02, 2022
Dani Kaplan This wasn't due to a change on 31st Dec. The problem is caused by an integer overflow error: the anti-malware component is converting the date/time into "YYMMDDHHMM" format and storing it as a signed 32-bit number (max value 2147483648). So, in Dec 2021, the number would start with "2112..." (below the threshold). In Jan 2022, the number would start with "2201..." (above the threshold).

The_Exchange_Team I think it would be useful to add this info to your blog post. I also have to echo what David_Richard said: when I was investigating the problem, I found Twitter threads (by your customers) several hours before anything appeared here. Even if you didn't have a solution right away, at the very least it would be useful for you to say "We know about this and we're working on it". That would save people time, e.g. I initially thought that it was a local problem to my environment so I was looking for anything related to back pressure.

In theory, this would be an ideal scenario for EEMS, if you could fix the problem automatically for any IT staff who finished work on Friday and won't find out about it until Tuesday morning. However, I'm guessing that this isn't possible if the fix requires customer action.