Slower SMB read performance for large files in 22H2
Published Oct 03 2022 05:35 PM 121K Views
Microsoft

Heya folks, Ned here again. There is a performance reduction in 22H2 when copying larger files from a remote computer down to a Windows 11 computer or when copying from any Windows computer to a remote Windows 11 computer. 

 

Fix update 03/06/2023: update February 28, 2023—KB5022913 (OS Build 22621.1344) Preview - Microsoft Support contains the kernel update that fixes this issue. 

 

For the latest information about this issue, please see the known issues section of KB5017389

If you are using Microsoft 365 admin center and have Windows Release Health available there, you can view this issue in the Windows 11, version 22H2 section. The ID on Microsoft 365 is WI442499.

103 Comments

Thanks for letting me know and bringing the temporary solution. :happyface:

Hi @NedPyle thanks for letting us know.

 

May I share some thoughts and questions? 

 

1. Could it be the "lazy-writeback", introed and enabled by default since 1809 builds could be a cause?

 

Even when not utilizing SMB over network but usual local copy jobs e. g. from / two nvme one might notice a massive jigsaw effect when copying solid bigger files, especially with File Explorer, not so much with Robocopy as of its nature of multithreading.

Unfortunately there is no flag such as ADMX / CSP to disable lazy write-back for Explorer, to revert to the default behaviour of Windows Client and Server versions before build 1809.

The fact unbuffered IO does help, could be indicating this. 

 

2. Could you exlude the now wider default use of SMB compression, to be a potential cause for the issue? 

 

Microsoft

I can't share too many details yet, but we know exactly where it is getting slower in Kernel, nothing to do with SMB or compression. 

Copper Contributor

I would just want to correct 40% slower statement from Ned, im seeing way over 100% difference on setups like 10gb NICs (5gb/s vs 2gb/s) difference, or on SMB multi channel setups with multiple gigabit NICs.

Microsoft

@ivicask Thanks, I'll make sure that's clear above. The variance can be quite large

Brass Contributor

Thanks for letting us know, @NedPyle!

Pedro

 

Copper Contributor

@Ned Pyle

Can this issue cause the copied data to arrive corrupted on the destination drive?

Microsoft

Not to our knowledge. In fact, it should make corruption less likely

Copper Contributor

Was this fixed in KB5019509?

Microsoft

@jfm666 No changes yet

Copper Contributor

Is Windows 10 22H2 equally affected or just Windows 11 22H2? What's the ETA of the fix? I am holding off upgrading Windows 11 systems to 22H2.

Microsoft

@guhoo Windows 11. I don't have an ETA from the team that owns this code, it's not my own. 

Copper Contributor

@guhoo  Tested on Windows 10 22H2 is not affected and copies full speeds as usual. 

Copper Contributor

What is the ETA for the fix? It's been 3 weeks already and this is a very high profile bug in Windows 11 22H2.

Microsoft

@guhoo I cannot share timelines, sorry.  

Copper Contributor

Even though I have excluded source and destination in windows security real-time protection, disabling it completely gave back most of the nic throughput.

 

Having this regression around for an entire month, is uh, disappointing...

 

 
Copper Contributor

I have Windows Security disabled. Transfer speeds are still cut by 50%.

Copper Contributor

My NIC performance is reduced at least by 50%.  WiFi unaffected.  Have tested on multiple Win 11 22H2 PC's, multiple routers, multiple branded NICs - same results. 

Copper Contributor

Correction.  WiFi performance is degraded as well.

Brass Contributor

slowdown SMB copy problem continues (as in not fixed), even with newly released KB5019980 update for Win11 22H2 as the problem is still noted there.

Copper Contributor

It shows a lack of concern for this flagship desktop OS.

Brass Contributor

Dear Microsoft team when will this severe bug be fixed ? Frankly this has taken too long to fix at this point. Please update us on when we can expect the fix ? Thank You

Copper Contributor

I think it's time for Elon Musk to acquire Microsoft to light a fire under them.

Deleted
Not applicable

Someone said once: "Feedback is a gift.". 

@Lance Longreen quality feedback vs @guhoo any helpful feedback.

Microsoft

See updated note in the blog post. The preview fix for this issue is in Windows 11 Insider Preview Build 25252

 

The permanent production fix will come in a normal monthly cumulative update after this is validated. As I've mentioned before, the fix is not in SMB, and since it's not a component I own, I don't have further details on timing. 

Good to hear. Thanks for the follow up Ned, much appreciated. 

Brass Contributor

thanks Ned.

 

though the fix didn't seem to make it onto the recently released 22621.1020 & 22623.1020 insider preview beta builds, just only with insider build 25252

Copper Contributor

Could this be affecting OneDrive sync? We recently migrated a client to OneDrive from their on premises file server and we had a report yesterday that it was taking hours to upload 60x 5MB photos. 

Brass Contributor

@erpmanila3w Where do you see that its fixed in insider build 25252 ?

Brass Contributor

@Lance Longreen , see "Fix update 11/28/2022" in Ned's post 🙂

Copper Contributor

SMB SLOW RATE TRANSFER FILE NOT SOLVE 

Brass Contributor

Does the last comment mean there is still a bug and it was not fixed in insider build 25252 ?!

 

Also when can we expect a mainstream non insider release of this severe bug ?

 

It's been far too much time at this point for something this severe.

 

@MOGAbyte  please share your usecase and reproduction steps. And your winver (Windows Build).

 

For me it is solved in release channel or other channels of Windows 11. Especially when writing to USB 3.x via UASP.

Copper Contributor

What does "release or other channel" mean? Latest public available build is "22621.963" and its not solved yet, this bug is know since early versions of 22H2 somewhere early in the the summer, so its unsolved for 6+ months now....

 

I never had any speed issues writing to USB, this was exclusively bug with network, and impact can be drastic in business environments, but everyone is affected just dont notice it necessary.

 

For example im copying 200GB VHDX, i have 2x 1gbit NIC, im getting around 70MB/s,  while im getting around  220MB/s on previous windows build.

 

That means i need to wait 3x times more in order to copy files..

Deleted
Not applicable

@ivicask it is a multi-layered issue. It does not only affect SMB this means also local file operation.

When you bring up teamed network it is getting more complicated. We have either LFBO Teaming or SET Teaming which work absolutely differently.

Can you actually uninstall a Windows CU and get the former performance?

Copper Contributor

I dont notice any speed impact on local files, im copying right now 2 files between 2 samsung nwme drives and its copying full speed that this drives are capable.

 

I dont remember if its even CU update which you can uninstall or you need to use option to go to previous version of windows, but it works fine on any 22H1 version, and not on any 22H2 no matter the updates/CU.

 

Brass Contributor

Multi layered issues also including local file copies now ?!

 

It has been over 6 months now.

 

Microsoft team and @NedPyle , when will this severe bug finally receive the attention it needs from your dev team and get fixed ?!

Brass Contributor

@Lance Longreen 

maybe in upcoming Jan 10, 2023 patch Tuesday update for 22H2 or newer for non-insiders

 

Does the last comment mean there is still a bug and it was not fixed in insider build 25252 ?!

no, it was already fixed in build 25252 & greater; just not yet fixed with 2262x (22H2)

 

I'm not sure if the recent insider preview 22621.1095/22623.1095 Jan. 5 builds include the fix

 

edit - just wait until Tuesday January 10, Lance (it's only ONE MORE day).  maybe MS will publicly release a fix by then

Brass Contributor

@erpmanila3w if its 1 day until patch then surely the patch notes are available by now ???? and we can see if its in this patch or are You just guessing that it might perhaps maybe be in a patch Tuesday ?

Copper Contributor

Installed 22623.1095 inside VM, its NOT fixed!

https://imgur.com/a/8EV9ia3

Copper Contributor

The update that came out today does not fix it. It is still listed as a known issue: https://support.microsoft.com/en-us/topic/january-10-2023-kb5022303-os-build-22621-1105-c45956c6-4cc...

Brass Contributor

Don't know what to say. The fact that something so trivial and at the same time very core, integral and important such as this has gone unfixed for plus 6 month in Windows by Microsoft says a lot, unfortunately not very good things. 

Yuck. Thanks for the confirmation @MikeG1765 + @ivicask. Anyone of you having unified (premier) support? I wonder why what the underlying change was to raise this regression.

Brass Contributor

@MikeG1765 

thanks Mike - the KB5022303 update from Jan. 10 (build 22621.1105) definitely does NOT fix this problem

 


 

Don't know what to say. The fact that something so trivial and at the same time very core, integral and important such as this has gone unfixed for plus 6 month in Windows by Microsoft says a lot, unfortunately not very good things.


@Lance Longreen 

yup.  how highly disappointing that the problem still remains unfixed.

 

gonna have to wait for either a late January or early February update

Copper Contributor

This is getting extremely frustrating. More so the fact that this has not been prioritized as an issue requiring an immediate hotfix and it's been going on for months now. And yes, it does affect local transfers as well as we can see on our workstations with Megaraid RAID6 SSD arrays. Not only it has not been prioritized but it is also being downplayed in the KB notes as something affecting few users when it is affecting everyone as it looks like a kernel issue.
And no we can not use robocopy. Not everyone just "copies files". A lot of us use applications over the network.
We are a small software engineering company and we heavily use very large VMWare Workstation images served over the network. The performance is crawling and we are resorting to copying images locally in order to work. Our entire workflow is messed up and we're spending 4/5 of work time, swapping images around. And we can not revert workstations to previous builds either because we need the latest WDK.

Just because the majority of Windows users do not notice the underlying problem, it doesn't mean this should not have been fixed yesterday. Do we really have to contact every major tech news outlet site informing them of the situation so this gains publicity and traction before Microsoft prioritizes a hotfix?

Brass Contributor

@Nodens  wrote:

Just because the majority of Windows users do not notice the underlying problem, it doesn't mean this should not have been fixed yesterday. Do we really have to contact every major tech news outlet site informing them of the situation so this gains publicity and traction before Microsoft prioritizes a hotfix?

 


NO, Nodens!  don't even think of going to that extreme,  sheesh!

heck, the problem won't get fixed until maybe either mid-February or in March so folks just come back next season when winter changes into spring (either late March or April) and wait patiently for the public official fix.  the recent 1/26 KB5022360 update for 22H2 (for non-insiders) still has the copy performance slowdown problem and MS only released new beta insider builds (22621.1245 & 22623.1245) {KB5022358} that possibly contains the fix but are still only available for windows insiders.

Copper Contributor

Thank you, Ned, for being forthright about the issue. I've tried every troubleshooting step I and network card configuration I could to get the same download speed over 10Gbe that I was getting with uploading. Nothing worked. Without this post, I would still be looking in vain for an answer, blindly trying out new settings.

Nevertheless, I cannot stress just how disruptive this bug is to the workflow in our company. Please work this out soon and release a public bug fix.

Copper Contributor

I understand that we're being told versions 25252 on up are fixed but this is not the case in my findings.  I will admit, it's better than 22621 and 22623 for sure but it's NOT fixed.  Going back to version 22598 has reminded me of how it SHOULD be.  Everything is faster, file transfers and web page loading to be exact, AND my laptops are FINALLY staying cool and getting MUCH better battery life, though they both have discrete video cards.  I tried using 25267 as a wired in file server, just SMB, and it's odd how it's speed fluctuates.  I'm used to system-to-system transfers of 300 on up average over 5Gb and sustained 120 over 1Gb.  This is NOT the case with any version after 22598.  I have my own theories as to what's being attempted here at the kernel level, but I'll keep it clean 🙂

 

This is just one person's ever-long experimentations and final conclusions with just a hint of disappointment at microsoft.

Brass Contributor

I had not joined this conversion until now because I trusted that Microsoft was actually working on a fix for this. However, I need to chime in here and say that this issue is NOT fixed in Windows 11 Insider Beta build 22623.1245, as claimed in Microsoft's release notes:

  • We fixed an issue that affected copying from a network to a local drive. Copying was slower than expected for some users

After fighting with this issue for almost 5 months now, I was eager to test out this fix in 22623.1245. Unfortunately, I'm still seeing very slow file transfers moving data from my 10GbE-connected server on this build, on the order of 50-60% of expected performance for gigabit clients and 30% of expected performance for 10GbE clients. 

 

That I found effectively no change in performance in this build was the last straw, so I made the decision to roll back several PCs with wired LAN connections to Windows 10. This was hours of my time that I did not need to spend. 

 

I get that most consumers on wireless LANs will never see this issue, but lots of businesses use Windows on wired LANs and depend on fast file transfers. We upgraded to 10GbE a couple years ago for better performance. Seeing this investment undermined by Windows 11 is truly disheartening.

 

If think if this had happened "back in the day", Dave Cutler, Moshe Dunie, or Jim Allchin would have fired the team responsible.

 

Please Microsoft, take this seriously.

Brass Contributor

Can also confirmed that its NOT fixed in both 22621.1245 and 22623.1245 (tested both just for the sake of it) this file copy bug is NOT fixed. To be honest this is a highly critical bug, i cannot believe ostd still not fixed even You write it is in your insider release notes. Microsoft it is really time to step it up ! @ned can we please get some official reply to why it's still not fixed even in latest insider builds?

Co-Authors
Version history
Last update:
‎Mar 06 2023 09:51 AM
Updated by: