SMB Compression: Deflate your IO
Published Mar 30 2020 12:00 AM 19.1K Views
Microsoft

Hi folks, Ned Pyle guest-posting today about SMB Compression, a long-awaited option coming to Windows, Windows Server, and Azure. As you know, the growth of cheap storage and plentiful network bandwidth over the past decade had an immediate effect: files got bigger and less efficient ;). SMB Compression tackles this by adding optional inline whitespace compression to file transfers, removing congestion and copy time from offenders like virtual machine disks, raw graphics, scientific data, and other large file formats. Depending on the (in)efficiency of file formats and the IO pattern, the performance increase in copies can be impressive:

 

2020-02-19_17-06-09.png

 

Everyone who’s ever seen a ZIP file knows what compression is, so let’s go straight to the demo and see it. Here I compare using robocopy with and without the new /compress parameter on a very compressible file, then try again between a very compressible and very un-compressible file.

 

 

As you saw, we also provide for the case of “what if I don’t know how well a file will compress? I don’t want to make my copy slower in order to find out.” The solution is we measure compression efficient for an admin-specified file size and if we aren’t seeing savings, we just turn compression off automatically.

 

SMB Compression uses a negotiation mechanism, so multiple compression algorithms are possible, and vendors can add their own. Currently the list is XPRESS (also known as LZ77), XPRESS Huffman (LZ77+Huffman) and LZNT1. This gives us excellent CPU performance bang for our compression buck. It’s not tied to dialects so much as OS versions – there will not be an SMB 3.1.2 to make this work. SMB Compression is already part of Windows 10 and the SMB2 protocol spec, although it currently has no usage options or server OS to make use of it.

 

The first usage options of SMB Compression will be:

 

  • Hyper-V Live Migration with SMB
  • Robocopy
  • Xcopy

We don’t have a user experience for Chad from Accounting, currently; our first phase will be for applications and admins like you. One thought is that we could enable compression on a per mapped-drive case and greatly lower the threshold of measuring efficiency, but we’ll be experimenting and asking you questions on this as the feature evolves.

 

The question I typically get at this point is: when is this coming? I don’t have a good answer yet, but as we get firmer, I will get more details out there. This is a key technology for Azure Files and Windows Server computing, so all I can say is that it’s coming. As you can see from the demo, we’re far along. Check back at the IT Ops Talk and FileCab blogs for more details and info on Insider Previews this year. We’re working with third parties to offer this option – ask your SMB vendors about their plans.

 

I hope you’ve enjoyed learning about this new feature, I think it’s going to help you save a lot of wasted time and bandwidth. If you have questions, hit me up on twitter or DM me on TechCommunity.

 

- Ned Pyle

10 Comments

Looks so promising. I am curious if this tech could be even incorporated into things like BITS, DU or Branchcache @Ned Pyle 

Microsoft

@Karl_Wester-Ebbinghaus anyone who implemented SMB as their transport can utilize this, so the answer is "yes, we just need to convince those teams with customer help" :)

Will you need the network card to have the technology?

Microsoft

@Gabriel Luiz nope, it will work with any sort of physical or virtual NIC except if you are using an RDMA NIC in SMB Direct mode. And we are working on adding that support too.

@Ned Pyle what could we do that robocopy get's better multithreading performance and cpu utilization.

Comparing to Gurusquad gscopy 360.

The difference in the field is amazing over SMB 10G, not sure if they use compress. 

 

And about WAC SMS does it use compression, there is no switch and on small files performance is like robocopy. 

@Ned Pyle the Server 2019 robocopy has no /compress option yet. Do you know when this will become GA?

 

Microsoft

@Karl_Wester-Ebbinghaus Next version of Windows. Can't share date yet

@Ned Pyle 

Good news it has been backported to

2004 / 20H2

1903 / 1909

1809 /Server 2019 no news yet.

 

https://www.borncity.com/blog/2020/09/18/windows-10-version-1809-190x-erweiterungen-bei-robocopy-mit...

Copper Contributor

Do I need to have you when get downloaded the software?

 

@yueyuetraveltoursinc not sure if I got your right, maybe you ask in your native language Edge will have my back. If you supposed to ask this: The feature is integrated into Windows 11 21H1 or later and Windows Server 2022 or later.

Version history
Last update:
‎Mar 29 2020 11:22 AM
Updated by: