How to improve Windows cumulative update installation times
Published Feb 18 2021 11:23 AM 40.8K Views
Microsoft

With the new ability to deploy Windows SSUs and LCUs together with one cumulative update, I wanted to offer some background on how cumulative updates grow and the best practices you can employ to improve the update experience.

Why and how do cumulative updates grow?

The longer an operating system (OS) is in service, the larger the update and associated metadata grow, impacting installation time, disk space usage, and memory usage. Windows quality updates use Component-Based Servicing (CBS) to install updates. A cumulative update contains all components that have been serviced since release to manufacturing (RTM) for a given version of Windows. As new components are serviced, the size of the latest cumulative update (LCU) grows. However, servicing the same component again does not significantly increase the size of the LCU.

In the image below, you can see two examples, each with three changes. The resulting LCU in example #2 will be larger than the LCU in example #1 because it contains components A, B, and C.

01_lcu-examples.png

During an OS release, the size of the update grows quickly and then slowly levels out as fewer and fewer un-serviced components get added to the LCU.

02_lcu-over-time.png

With regard to the graph above, you'll see that there are points where the size goes down briefly due to a specific edition of Windows going out of service. For example, the Windows 10 Enterprise Long Term Servicing Channel (LTSC) has a longer servicing lifecycle than Windows 10 Home edition. When the Home edition reaches end of service, we stop servicing any components in the Home edition, which reduces the size. Shared components still get serviced.

Starting with Windows 10, version 1809, Microsoft implemented new technology to reduce the size of the LCU and growth rate, and improve installation efficiency. Maliha Qureshi’s blog post What's next for Windows 10 and Windows Server quality updates coupled with the Windows updates using forward and reverse differentials document, aim to help you understand more about this change to support forward and reverse deltas and the benefits of reducing the size of the LCU.

Starting in Windows 10, version 1903, more improvements came along in the update package. For example, if individual binary files in a component are unchanged, they are replaced with their hash value in the update package. In the example above, if Component A had five binary files and four changed, we would only send the payload for the four that changed. Overall, this results in the LCU growing about 10-20% more slowly and is useful when the image or video files are part of the component and rarely change.

What happens when an update is installed?

The Windows quality update installation process has three major phases:

  • Online: The update is uncompressed, impacted components are identified, and the changes are staged.
  • Shut down: The update plan is created and validated, and the system is shut down.
  • Reboot: The system is rebooted, components are installed, and the changes are committed.

The shut down and reboot phases affect users the most since their systems can’t be accessed while the process takes place.

Best practices to improve installation time

Now let's get to best practices that can improve your update experience:

  1. If possible, upgrade to Windows 10, version 1809 or newer. Version 1809 offers technology improvements that reduce the size of the updates and more efficient to install. For example, the LCU of Windows 10, version 1607 was 1.2GB in size on year after RTM while the LCU in version 1809 was 310MB (0.3GB) one year after RTM.
  2. Hardware optimizations:
    1. Run Windows and the update process on fast SSD drives (ensure the Windows partition is on the SSD) instead of HDD by placing the Windows drive in the SSD. During internal testing, we’ve seen up to a 6x install time reduction from SSD vs HDD.
    2. For systems with SSD (faster than 3k IOPS), CPU clock speed is the bottleneck and CPU upgrades can make a difference.
  3. Antivirus (or any file system filter driver): Ensure you are only running a single antivirus or file system filter driver. Running both a 3rd party antivirus and Microsoft Defender can slow down the update process. Server 2016 enables Defender by default, so if you install another antivirus program, remember to disable Defender. The fltmc.exe command is a useful defrag tool to see the running set of filter drivers. You can also check out virus scanning recommendations for Enterprise computers that are running currently supported version... to learn which files to exclude in virus scanning tools.
  4. Validate that your environment is ready to apply updates by checking out the Cluster-Aware Updating requirements and best practices.

Best practices to improve performance of Windows Update offline scan

Using Windows Update Agent (WUA) to scan for updates offline is a great way to confirm whether your devices are secure without connecting to Windows Update or to a Windows Server Update Services (WSUS) server. WUA parses all cumulative update metadata during a scan so larger cumulative updates require additional system memory. Cumulative metadata can become quite large and must be parsed in memory for WUA offline scans so machines with less memory may fail the WUA offline scan. If you encounter out of memory issues while running WUA scans we recommend the following mitigations:

  1. Identify whether online or WSUS update scans are options in your environment.
  2. If you are using a third-party offline scan tool that internally calls WUA, consider reconfiguring it to either scan WSUS or WU.
  3. Run Windows Update offline scan during a maintenance Window where no other applications are using memory.
  4. Increase system memory to 8GB or higher, this will ensure that the metadata can be parsed without memory issues.

Microsoft continues to investigate and improve the Windows 10 update experience as it is a critical component to stay productive and secure. Use the Feedback Hub or contact your support representative if you face any challenges when updating.

 

21 Comments

Before anyone gets to wrong conclusion of this very helpful blogpost. 

 

The CU size alone is not the reason why your Server 2016 1607 LTSC or Windows 10 2016 LTSC takes ages to get patched and waiting at restart compared to Server 2019 1809 LTSC or Server vnext "2022" 21H2 LTSC or later Windows 10 (LTSC) 

 

The main reason for this was eliminated in the build of 1703 which is not a Server LTSC release.

 

If you are interested into details, live view and comparison check out my sufficient benchmarks, videos in the comments here 

https://windowsserver.uservoice.com/forums/295047-general-feedback/suggestions/32121229-stop-the-win...

Another point for slow CUs are commonly Laptops with HDDs. 

 

Why is that :

In the field I see very high HDD fragmentation rates of 4% to up to 30% or even more.

 

A fragmentation of 3% is enough to severe decrease overall Windows response, not limited to CU install times.

A System with 5% or more can take ages to do the most simply task just like installing VLC taking 5 minutes while task manager reports high HDD usage of 50-100%.

 

Other factors :

Notebook CPUs are really slow especially old ones from AMD and Intel (Dual / Quad) from 2006-2014

Slow means they are not comparable to Desktop CPUs at all.

 

HDDs in Notebooks are small 2.5 drives with few cache and 5400rpm or less and thus having tremendously bad random access times, which is crucial for CBS/DISM/Tiworker.

 

I am not saiyng old laptops or computers are to be replaced. But a new SSD and more RAM can do wonders. 

"But why Laptops HDDs are so highly fragmented in the first place?"

 

The answer has several layers

Laptops are Optimized for portability, low energy consumption (battery lifetime) usually not for performance

 

Slower CPUs, slow small form factor harddisk and most of all USECASE.

 

Laptops are often not online for several hours. Standby / shutdown of users prevent defrag from running at all with a vicious circle of the fragmentation getting higher and higher and taking longer and longer to complete which again doesn't comply with short usage or short absence time to run sufficiently.

 

So what's again the solution upgrade the HDD by SSD. 

You won't regret the effort to clone the OS for the same device on a different OS disk.

 

Microsoft

Great suggestion for older laptops K_Wester-Ebbinghaus.  SSD truly is a huge upgrade for day to day work, and for installing updates.

Brass Contributor

The v1607 updates continue to get bigger. Now at 1.6 GB. Tough on some networks.

How about announcing that by a certain date, you must have that update installed before you get further updates which would be just the difference between that date and the current fixes.

For example, in order to get the latest updates you must have the February 2021 update installed. All updates after February 2021 would be based on the difference between February 2021 and the current date.

 

The idea is not that bad. Based on practice this will lead to Patch fragmentation which is the main reason for CUs compared to anything before Windows 10 /2016

 

I also doubt that the servicing Stack is so flexible for this change. Otherwise the including of SSU into CUs could have been made for all releases. 

 

Mind that the major drawback of LTSC for Server an Client is basically the fact it is feature frozen zone. 

Microsoft

Edward_B, Thanks for the suggestion.  We are prototyping that idea and a few others, although as K_Wester-Ebbinghaus mentions any plan has to fit in the constraint of LTSC and the servicing stack features that were shipped with 1607.

Copper Contributor

Hello All,

In this month (February 2021), we have seen that unless SSU gets installed on the Windows server 2016 CU patch wasn't showing up in the software center this kind of scenario, we haven't seen till now.

Last SSU which got installed on the servers was in Sept 2020 and till Jan 2021 servers were picking up CU even though every month SSU was made available via SUG.

Any thoughts on this one.

@Aswani Kumar Griddaluri can you please check if you have deployed this SSU
Microsoft Update-Katalog

 

Brass Contributor

Reading this thread it seems the issue with update restarts on Server 2016 is 'fixed' in 1703 and above but not backported to Server 2016 (1607) because it's a LTSC release and doesn't get new features. 

 

What about LEDBAT?  They backported LEDBAT to Server 2016 but can't backport a WU reboot fix?  IMO, this is less of a new feature and more of a bugfix that affects thousands of customers.

 

And for everyone saying to just upgrade to Server 2019 or 2021 later this year?  Easier said than done.  We just got rid of our last Server 2008 R2 box last year.  Server 2016 will be around in datacenters for many years to come.

 

 

Microsoft

Aswani,
Sometimes we have to make an SSU a pre-requirement to get future LCU's due to the type of issue fixed in the SSU.  We generally try to avoid it, so not every SSU is considered a pre-req.

We do have technology coming soon that will bundle the SSU with the LCU into a single .MSU file.  The first version of this actually ships for Windows 2004 and 20H2 tomorrow.   Here's the blog post that announced the technology:
Deploy Windows SSUs and LCUs together with one cumulative update - Microsoft Tech Community

The technology will come to 1607 in the future as well.

Mike

Microsoft

Cristopher,
As you mention with the LEDBAT example, backporting features to an LTSC version of Windows can be supported.  Our goal with LTSC though is to keep it as stable as possible and not introduce features that would be risky.  The servicing stack has gone through many significant architectural changes from 1607 to 1703 (and even more to 1809) so backporting any features in the servicing stack is quite risky.

Thanks,
Mike

Copper Contributor

@Karl_Wester-Ebbinghaus 

That's true. Post-installing the SSU and initiating machine policies on the server, we were able to see the CU on the server.

 

@Mike Benson 

Thanks for the update.

 

regards,

Aswani

Glad to see it helped @Aswani Kumar Griddaluri 🙏🏽

Silver Contributor

Thank you for sharing.

Changing the hard disk from HDD to SSD is really challenging and since it normally means complete reinstallation of Windows and all applications, companies ignore it.

However, it is always recommended like when companies are purchasing new device, make sure it is running SSD.  

changing from HDD to SSD is really challenging

can you elaborate why?
PC: Usually one adds an addional SSD and clone the HDD to SSD. There are different methods, even DISM would work but is more complicated than competitor products

Notebook: backup the HDD to an external HDD or SSD and restore to the new SSD.

 

I agree that many companies do not do this, especially not for Notebooks as they are mostly hard to maintenance and need a lot of time to open and close. Some notebooks have such a silly layout that you have to remove the whole mainboard to get to the HDD or RAM so from a cost perspective it might be even better to buy new ones. 

Copper Contributor

Hello.

Just a question regarding the monthly CUs for the Servers.  (Server 2016 and Server 2019)

Do they need to be installed in sequence?

For example, if I install January's CU and cannot install CUs for two months.  Now it's April.

Can I simply install the CU for April or do I need to install February and March CU first?

 

Thanks.

Microsoft

cmattali,

Since they are cumulative you can just install the latest and it will ensure you also get all previous changes.

 

While it is possible to skip months, I wouldn't recommend it since there are important security updates every month and your servers are most likely critical infrastructure.

 

thanks,

Mike

Copper Contributor

Thanks for the response.

I have a system that is very difficult in obtaining a downtime window.

I wanted a safe method of installing the CUs in the event some months were skipped.

Just to confirm.  It's safe to install the latest CU even if previous CUs were missed?

 

Microsoft

yes, it is safe to install even if previous CU's were missed.

 

thanks,

Mike

Copper Contributor

Thanks.

Co-Authors
Version history
Last update:
‎Feb 02 2023 10:12 AM
Updated by: