Exchange Server Mail Stuck on Queue due to Microsoft Bug 01/01/2022

[SABBIRRUBAYAT]

There is a live Bug going on in On premise Exchange. To resolve the issue Please Disable your Antimalware filter just for now so you can start your communication again. Please note that this is not final solution. this is just workaround for now

Please follow the below step:

1. Go to This location “C:\Program Files\Microsoft\Exchange Server\V15\Scripts\” and run Disable-Antimalwarescanning.ps1     Follow Microsoft Article if you need help to run the command https://docs.microsoft.com/en-us/exchange/disable-or-bypass-anti-malware-scanning-exchange-2013-help
2. Restart Transport Service
3. New-SettingOverride -Name "DisablingAMSIScan" -Component Cafe -Section HttpRequestFiltering -Parameters ("Enabled=False") -Reason "Testing"

Following third step is not necessary. This step is necessary only if you have third party malware agent. 

[#2]

I confirm this has allowed for communication flow to resume

[#3]

@SABBIRRUBAYAT Where do you see that there is an 'Cyber Attack' ?

It seems there is a bug in the moduoe 'FIP-FS' if you look in your Eventlog

The FIP-FS "Microsoft" Scan Engine failed to load. PID: 20132, Error Code: 0x80004005. Error Description: Can't convert "2201010004" to long.

Looks like there are using long in there application
with an max number of
214748 3647 and they try (date looks like?) to parse and higher number 2201010004

A solution without a reboot from servies is:

Set-MalwareFilteringServer -Identity <<SERVERNAMEHERE>> -BypassFiltering $true

Please note!! This is a temporary solution! get a good virus, mallware and spam scanner on your Exchange server from a third party!

[#4 · Accepted solution]

@amaltsev Are you sure this is a cyberattack and not a bug?  According to most reports I've read, this is due to a failure converting to a new date and occurred at midnight UTC.

[#5]

It's a bug!
According to @ceno666 the issue also seems to occur with the 220101002 update version as well. Could be related to, what I’m dubbing, the “Y2K22” bug. Refer to the comment from JulianSiebert about the “signed long” here: https://techcommunity.microsoft.com/t5/exchange-team-blog/december-2021-exchange-server-cumulative-u... The “long” type allows for values up to 2,147,483,647. It appears that Microsoft uses the first two numbers of the update version to denote the year of the update. So when the year was 2021, the first two numbers was “21”, and everything was fine. Now that it’s 2022 (GMT), the update version, converted to a “long” would be 2,201,01,001 - - which is above the maximum value of the “long” data type. @microsoft: If you change it to an ‘unsigned long’, then the max value is 4,294,967,295 and we’ll be able to sleep easy until the year 2043!

Paolo Caparrelli
GOLINE SA
IT Services

[#6]

@SABBIRRUBAYAT - just a note that it's not a cyber attack, it's an int32 conversion issue with the date code of the 2022 signatures that Microsoft still hasn't fixed.

I had also originally disabled malware scanning using the disable scripts last night to get things working, but the problem with this is that the malware definitions/engines are never updated for the FIP-FS service when you do that...it's better to set all your servers to bypass scanning instead.  I had tried this last night before the internet blew up (this is for 2013 but same thing for 2016/2019 - https://docs.microsoft.com/en-us/exchange/disable-or-bypass-anti-malware-scanning-exchange-2013-help).  The problem with that article is it doesn't say to restart the transport service after running the bypass command - but you have to do that for it to take effect. 

If you do it this way, mailflow will work and your definitions will continue to be updated so that hopefully when Microsoft fixes this your servers will be updated so that when you remove the bypass things continue to work (rather than re-enabling scanning, then waiting for the engines to update which takes a long time/breaks your mail flow again).

[#7]

I only ran the two steps below, and the email started to flow after.
1. Disable-Antimalwarescanning.ps1
2. restart the transport service
I did not run Set-MalwareFilteringServer <ServerIdentity> - BypassFiltering $true instead of the above two.

Will the next Exchange patch release installation re·vert
the antimalware agent back "on," as on the default installation, and again we have to disable the agent?

[#8]

It’s one or the other, not both - I personally recommend bypassing instead of full disable so you’re still receiving updates. Your malware scanning will stay on bypass for each exchange server until you set it back to disabled and restart the transport services again. But it’s entirely up to you on what you do…

I can’t believe Microsoft support hasn’t released anything official about this at this point.

[#9]

How funny that IF this was a cyber attack (which it is not) that your recommendation is to bypass/disable filtering.

If it was a cyberattack, I think you’d want to leave filtering enabled even if it meant no mail flowing for a while…

[#10]

@Corbett Endersmy thoughts exactly. however I stumbled on the fix from an old article that recommends bypassing it. I've also kept an eye on the firewall, and nothing out of the ordinary.

[#11]

It looks like MS has released its fix!, is it official?

https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/30494...

[#12]

Thank you , saved me to search this morning. (Can confirm this works)

[#13]

I apologize . this is a bug

[#14]

Definitely you are right. I found the solution at the first hour and I started getting call from multiple companies at the same time. I was a little bit freaked out and thought this might be a cyber attack. I apologize for my mistake

[#15]

Hopefully yes

[#16]

Apologize for my mistake identifying as cyber attack . Actually the behavior pattern was like that . anyway lucky it was a bog not an attack

[#17]

Good Idea

[#18]

Thank you ..................... and apologize to identify this as cyber attack . thanks anyway

[#19]

@LeeMEI please note that my answer relates only to the action steps proposed, nothing to do with the origin of the problem