Known Issue Rollback: Helping you keep Windows devices protected and productive

[Eric_Vernon]

Today I'm sharing details about Known Issue Rollback (KIR), a new capability that can quickly return an impacted device back to productive use if an issue arises during a Windows update. In this blog, my engineering partner, Vatsan Madhavan, and I will walk you through this new capability.

In today’s digital world, we you know you rely on trustworthy systems and services to stay ahead of potential security threats to your enterprise environments. At the same time, you strive to sustain the productivity of your work forces. We know you need both: security and productivity.

The Windows Servicing & Delivery team works to help keep Microsoft customers protected and productive by continuously building and delivering updates to the intelligent cloud and intelligent edge. Over the past year, we have been working on improvements that help you ensure uninterrupted productivity even when rolling out critical updates to prevent or address potential issues.

Introducing Known Issue Rollback

Known Issue Rollback is an important Windows servicing improvement to support non-security bug fixes, enabling us to quickly revert a single, targeted fix to a previously released behavior if a critical regression is discovered.

Built in direct response to your feedback, the pieces that make Known Issue Rollback work came together in a functionally complete system beginning in Windows 10, version 2004. Every month, we release monthly updates with many of quality changes “contained” using the Known Issue Rollback capability.

While Known Issue Rollback was originally designed for user-mode processes, we have made phased improvements over the last year to the OS kernel and the boot loader to support this capability in kernel mode. Some versions of Windows 10 prior to version 2004, for example versions 1909 and 1809, have partial support for Known Issue Rollback built into the OS and we leverage that support whenever possible when shipping updates for those versions.

At the code level

When Windows developers code a non-security bug-fix, they keep the old code intact and add the fix.

The Known Issue Rollback infrastructure in the OS provides developers with a method that evaluates a policy to determine the execution path. This policy tells the OS whether a fix should remain enabled or not. If the policy states that the fix is enabled, then the new code runs; and if the policy says that the fix is disabled, then the OS falls back to the old code-path.

Today, fixes in our monthly updates are enabled by default -- i.e., the old code is disabled, and the new code is enabled. If a fix turns out to have a serious problem, Azure hosted services and Windows work in tandem to update this policy-setting on the device and disable the problematic fix. Enterprises will be able to exercise control over this policy.

Note: As mentioned earlier, we only use Known Issue Rollback with non-security fixes. Using this coding scheme retains the old code. In the context of security fixes, older code is typically more vulnerable or exploitable; this is why we don't use Known Issue Rollback with security fixes today.

How Known Issue Rollback works for the end user

When Microsoft decides to rollback a bug fix in an update because of a known issue, we make a configuration change in the cloud. Devices connected to Windows Update or Windows Update for Business are notified of this change and it takes effect with the next reboot.

Microsoft Privacy Statement on Windows Diagnostics

Once this happens, the Know Issue Rollback infrastructure will start reporting that the fix - the new code that has a problem - is no longer enabled. From this point on, the OS will fall back to the previous code that had a bug albeit a much more benign issue than the new code that has a problem.

While these devices would still require a reboot, in most cases we have identified and published a rollback before most end user devices would have had the chance to even install the update containing the issue. In other words, most end users will never see the regression!

Devices that have opted into providing Microsoft with diagnostic data then send very scoped information about which code path is being exercised. This data helps us learn how well the rollback is succeeding in the ecosystem.

Putting enterprises in control

Enterprise devices are typically behind a Network Address Translation (NAT) and a firewall, which means they tend to be part of an Active Directory forest and are often managed using Group Policy. For a Know Issue Rollback, Microsoft publishes a specific Group Policy on the Download Center that can be used to configure and apply a rollback policy (rolling back the code in the latest cumulative update or LCU) within an enterprise. A link to the Group Policy is included in the Windows Update KB article and release notes as mitigations for a “Known Issue.”

Microsoft Privacy Statement on Windows Diagnostics

In the KB article, we describe the issue and related information to help you and your IT administrators make informed choices. Our customer service teams are also aware of the Known Issue Rollback system and will be able to work with customers to identify problems with monthly updates and in turn coordinate a rollback if necessary.

Similar to the end-user scenario, devices that have opted into providing Microsoft with diagnostic data send specific information about which code-path is being exercised. This data helps us learn how well the rollback is succeeding in the ecosystem.

An example of Known Issue Rollback in action

We have been using the Known Issue Rollback since late 2019 to contain non-security fixes. Today, about 80% of fixes shipped on Windows 10, version 2004 (and later in-market versions), ship using Known Issue Rollback containment. Like most mitigation solutions, the real value of the Known Issue Rollback does not become truly apparent until you need it, and it works! Here’s an example.

The April 2020 preview non-security release (KB4550945) for Windows 10, version 1903 had a regression in a fix for the Microsoft Store. For gamers who had purchased in-app content (like PC game add-ons) from the Microsoft Store prior to the April update, that in-app content would no longer appear licensed once the OS was updated with this release. The net result was that the games would not run.

Fortunately, the Microsoft Store team used Known Issue Rollback.

On April 27^th, a pattern indicating that a regression might be emerging was noticed by our social media monitoring team. With just over 110,000 devices updated, around a dozen tweets/reddit etc. posts indicated the following:

• A consistent description of the problem (“Trial Period”, “App won’t run”, “App crash”, etc.)
• Keywords that suggested that a Windows Update may be at fault (“KB4550945”, “1903”, “Latest Windows Update” etc.)

Internally, we created a post-release incident report to research, identify, and confirm the root cause. By April 29^th, 170,000 devices had installed the April update and we had pinpointed the code change in the release that was problematic.

At 7:40 PM PST on April 29^th, we made the decision to initiate a Known Issue Rollback. By 10:00 PM PST, we had kicked off the Known Issue Rollback process through our Azure configuration service and began configuring Windows 10, version 1903, devices to remedy the Microsoft Store issue. By 6:00 PM PST on April 30^th, 145 million Windows 10, version 1903 devices had been configured to roll back the Microsoft Store regression. By May 3^rd, after only 72 hours, 236 million devices had been rolled back.

In this example, we were able to deploy a Known Issue Rollback mitigation within 24 hours of identifying the root cause of a problem. The result is that the overwhelming majority of Windows users never saw the regression. For them, the problem never existed; their machines quietly updated and there was no visible issue.

The Known Issue Rollback lifecycle

Known Issue Rollback configurations have a limited lifespan—a few months at most—because we expect to solve the underlying problem quickly and re-issue the fix. Once the underlying problem has been fixed, the Group Policy has outlived its usefulness. It becomes a benign setting and can be undeployed safely.

It’s worth noting that each Known Issue Rollback Group Policy is unique to a specific issue, i.e. a regression, and, as such, these policies are not cumulative in nature.

Conclusion

Today, Windows runs on over a billion devices spanning industries and countries around the globe. At Microsoft, we have been cultivating a service mindset in our journey to continuously improve our engineering and software update experience.

With Known Issue Rollback, your organization can remain secure and productive. We understand that the size of our ecosystem and the scale of our service requires a service maturity mindset. Known Issue Rollback is a great capability that can enable you to quickly recover from regressions without the risks associated with rolling back critical security protections.

Vatsan and I are happy to be on the team leading these innovations to keep you, our customers, secure while you also stay productive.

To learn more

For more information on Known Issue Rollback, watch this video:

[Vadim Sterkin]

Hi @Eric_Vernon!

Thanks for the blog post, it's nice to have a word from the PG on top of personal guesswork based on the MSKB article.

I've got a couple of questions.

1. Is KIR related in any way to the Windows Feature Store / Windows Notification Facility?

2. Is KIR related to another approach of fixing issues via the WU troubleshooter? E.g. https://support.microsoft.com/kb/4570336 Can you shed some light on this method anyway? E.g. whether it's still in use, when it's used instead of KIR, etc.

Thanks,
Vadim

[Eric_Vernon]

Hi Vadim,

Thanks for your interest in Known Issue Rollback (KIR) and your questions. KIR is somewhat related to the troubleshooters from the standpoint that both are trying to solve customer issues. Troubleshooters tend to focus on a class of issues such as Activation, Networking, Microphone etc. Known Issue Rollback is much more targeted however; it deals with a specific issue that is a result of a specific code fix that shipped in a monthly Latest Cumulative Update (LCU).

We are not sharing other information about the internals/implementation details about KIR at this time.

Thanks,

Eric

[will nimmo]

Eric, will known issue rollback also be available on the windows server platform,  or for other Microsoft products such as Office or .net framework?

[bdam55]

@Eric_Vernon: Great stuff.

Like @will nimmo I'd be interested to know if this is a thing that will impact the server OS where the criticality is often a bigger deal.  Though you could argue that on the server side there's probably less of a chance that a non-security update breaks critical functionality.

The other feedback I'd give is KIR GPO bloat.  I can promise you that most orgs are just going to add a KIR to the policy and never think of it again.  While on the individual level that's probably not a huge problem, if each KIR is it's own policy/whatever that you're publishing then overtime that's going to create bloat that will impact GPO processing times.  GPO is the poster child for 'set it and forget it' until a new admin comes in and asks what these 2000 policies do and why they were set.

[Vatsan_Madhavan]

Hi @will nimmo and @bdam55 ,

Thanks for your interest in KIR and your questions.

This technology works on Windows Server the same way as it works on Windows Client Editions i.e. newer versions such as Windows Server 2004 have fully functional support for KIR today while older versions have limited support.

At this time, .NET Framework and other Microsoft products like Office do not have support for Known Issue Rollback.

The concerns you've raised about GPO bloat and the consequent impact on processing times is valid @bdam55  - thanks for bringing these up! This is something we hope to address in the future as we continue to make improvements to KIR.

Thanks,

@Vatsan_Madhavan 

[bdam55]

Thanks @Vatsan_Madhavan, that's great info!  I should have specified though: can you describe the support for Server LTSC (2012, 2016, and 2019)?  I'm sure there's large farms of SAC out there somewhere but I've not seen them in the enterprises I interact with.

"Not happening" is a perfectly fine answer and kind of what users signed up for but I'd love to be surprised.

[Vatsan_Madhavan]

Thanks for the clarification @bdam55. 

As I mentioned previously, there is partial support for KIR in versions older than Windows Server 2004.

Specifically, we have some support for KIR in each of Windows Server 2019 LTSC (1809) and Windows Server 2016 LTSC (1607), though the extent of capability in each version varies (in general, newer versions of the OS have more refinements/capabilities). There is no support for KIR in Windows Server 2012.

[ron S.]

Point of clarification. Is this feature is limited to Windows in a business setting? 

Are Windows Home users are left out in the cold, with broken machines?

[Eric_Vernon]

@ron S. This is not limited to the business setting. Group Policy is the KIR business setting solution. For other versions of Windows outside of a business setting AND are connected to Windows Update, KIR is implemented from our cloud service.

[Brian_Klish_work]

Great timing.  I'm sure this isn't the right place for a request like this, but I'm not sure where else to request it.  Maybe report as an issue on M365 Admin Portal?  Can we can a KIR policy for this issue in KB5000802?  It's been a known issue for several days and still no fix.  The barcodes on our college ID cards are not getting printed until I uninstall this update.

After installing updates released March 9, 2021 or March 15, 2021, you might get unexpected results when printing from some apps. Issues might include:

• Elements of the document might print as solid black/color boxes or might be missing, including barcodes, QR codes, and graphics elements, such as logos.

[WaltReed]

@Brian_Klish_work 

Unfortunately Microsoft doesn't have a mechanism for handling feedback from IT professionals, they want us to use the Feedback Hub too, which is about the same as writing a note on toilet paper and flushing it. My feedback would be that Microsoft needs to up their game in regards to testing updates. The past several years have proven that their current testing process is totally inadequate and demonstrates incompetence at the highest levels. There is not a day that passes that I'm not fighting a bad Windows (or Office365) update somewhere. I'd like to sugar coat this, but the there is not enough lipstick produced on the planet to make this Microsoft pig look good.

[Eric_Vernon]

@Brian_Klish_work Sorry to hear you are having issues with printing. Take a look at the Windows Release dashboard (https://aka.ms/windowsreleasehealth) for details of the issue and when the fix will be released. Unfortunately in this particular instance, KIR could not be used.

[Susan Bradley]

Help us understand when KIR can and cannot be used so we know when (or if) to expect things to be handled in this manner?

[Susan Bradley]

P.S. The dashboard at this time just says you are working on the fix so there is not ETA at this time of when to expect the fix for the image printing problem and the Dymo label issue so at this time the only workaround is to uninstall the update.

[Eric_Vernon]

@Susan Bradley Thanks for the question.

There are several considerations for when KIR can be used (today):

• Used only for non-security fixes.
• More recent platforms, Windows 10 2004 and newer, leverage KIR more fully. Older platforms have limited KIR support.
• Each fix is evaluated by the developer prior to release to see if KIR is feasible.

Today we don’t publish KIR usage for each fix that goes into a monthly update (LCU). KIR is part of a larger strategy to improve customer confidence in the LCU. The callout to the Windows Release dashboard is to highlight that this is where Microsoft will communicate - not only information about issues but also whether an issue is solved with KIR. KIR-mitigated issues typically release much more quickly than updates to solve regressions, and we are working to expand KIR usage in Windows platforms.

[Susan Bradley]

@Brian_Klish_work 

Another out of band update out tonight:

https://docs.microsoft.com/en-us/windows/release-health/windows-message-center#1574 I think your fix might in that

[Reza_Ameri]

Thank you for the post, it is very valuable , however I see one minor issue here. It will work if update has been installed and Windows have been boot and while inside the Windows environment it is able to send a diagnose report , however consider the case where update affect the Network Driver or Reporting Mechanism so no report will be sent or the update caused failure in booting of the operating system where no report will be sent too.

In this case we are in state where systems are not reporting and it might be valid like user turn off their device or they are not connected to the internet or there is network connection issue (internet failure and so on) or it could be a failure like update failure. I am wondering is there any solution for such scenarios too or we should leave it to the expertise of the administrator to identify whether not reporting status of the update is due to failure or update installed successfully with no issue and there is only a connectivity problem. 

[Harjit Dhaliwal]

@Eric_Vernon In your response to @Brian_Klish_work, when he asked whether KIR is in effect for the current printing issues from the March 2021 patches, you mentioned "Unfortunately in this particular instance, KIR could not be used", however in your video in this blog at around marker 00:30, you describe the issue with a printing issue caused by a patch in September 2020, and KIR came to the rescue. So, why doesn't it work for the current printing issues? 

[Eric_Vernon]

@Harjit Dhaliwal Each fix is evaluated individually based on the component, the risk of the change, fix type, the platform etc. In the case of the print issue mentioned by Brian, this was a security fix that had issues and KIR currently only supports non-security fixes.

[Andres Pae]

Hi @Eric_Vernon 

So in AD environment - we should check regularly release notes to see if some regression  is confirmed , check does it affect our environment and if yes then apply related GPO? 

Another question - approx when it will Intune include this functionality? Azure Arc (for Servers)?

[Eric_Vernon]

@Andres Pae You are correct about your AD environment. If you are not seeing the issue, there is no need to apply the GP.

We are working on integration with Intune and other management solutions but I don't have an eta to share at this time.

[phimo]

I think this article should add information for consumer devices, considering a lot of home end-users like me learned about KIR because of the bug which happenned with KB5001330. I found this page thanks to the following article: https://support.microsoft.com/en-us/topic/april-13-2021-kb5001330-os-builds-19041-928-and-19042-928-... which I found on PCGamer.

[another_jeansagi_or_the_same]

Just today, 24apr2021, I learnt about KIR and the KB5001330 debacle.

For what I've read I'm concluding KIR hasn't acted upon my system. I mean I don't have this on my regedit:

HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \Control \ FeatureManagement \ Overrides \ 4 \ 1837593227

I'm on Windows 10 (Pro) 20H2 (19042.928, Windows Feature Experience Pack 120.2212.551.0), so KIR should have acted upon my system, but as I said it seems it is not:

For that reason, I'm asking:

How do I enable KIR on my system? and

How do I know a fix for KB5001330 is correctly applied to my system?

Kind regards,

J.

[Eric_Vernon]

@phimo Thanks for the feedback. There is a section about how KIR works for Consumer devices. To simplify the process, we push a change to the cloud and is reflected on an Internet connected client usually within 24 hours. It does require a reboot most times.

@another_jeansagi_or_the_same There should be no action you need to take except for a reboot. 

[phimo]

@Eric_VernonThanks (Merci! I'm French Canadian)!

The one thing that isn't clear about KIR and KB5001330 is whether everyone will get the KIR prompt or only some of us...

Are we 100% sure that everyone in need of a rollback will indeed get the rollback? What if someone doesn't want to take any chance and want to use the KIR?

[another_jeansagi_or_the_same]

> @jeansagi There should be no action you need to take except for a reboot. 

So, users are basically blind on this fix?

I mean there is no other way to know that a KIR fix for the KB5001330 is in place apart than an obscure register in regedit?

J.

[Nicholas_Steel]

I've configured Windows Update to prompt me to download updates and to neither automatically download nor install updates. This KIR system bypasses my configuration of Windows Update and makes automated changes to my system without my knowledge which is very upsetting for me.

[GerritEllmer]

We have observed the same thing here on individual clients. KB5001330 has been installed on clients by itself although we have deactivated all automatic updates via GPO. We use a 3rd party patch management. This has noticed that an update has been installed which has not yet been released internally.

How can we ensure that only we and not Microsoft can distribute updates directly to the clients? What mechanism did the client use to detect and download the update? As I said, "Check for Updates" and "automatic Updates" are deactivated.
And how did it happen that only a few clients received this update directly?  

[Eric_Vernon]

@Nicholas_Steel  & @GerritEllmer 

There are a couple of topics that you are bringing up so let me try to address them. 

Today KIR is a separate service from Windows Update. KIR only makes configuration changes as noted in the blog. It does not push Windows Update packages to machines. If a WU package was pushed without your knowledge to your environment, pls contact Microsoft Support to see how to get that addressed and/or send feedback through the Feedback application. This should not happen.

To answer the question of going around your Windows Update restriction, because it is a separate service, it is not controlled by Windows Update settings. KIR targets all machines for a Windows version(s) that is either having the issue or could have the issue regardless of whether the actual update has been installed eg. all Windows 10 version 2004 machines. This means that if a user has the update KB in question installed and is experiencing an issue, KIR will mitigate that issue i.e. the user will stop seeing problems after a reboot. If a user has not YET installed the update KB, the KIR configuration will sit dormant on the machine until the update is installed at which point the issue will be automatically mitigated without the user ever knowing there was a problem.

We are investigating the creation of some UI to help users know when the KIR has been configured on the machine. We don't have this capability today unfortunately.

[rocifier]

When I started reading this article I thought "What a fantastic idea!" but by the time I got to the end I felt very sad. It seems that the great power of such a feature is that consumers would be able to pick which individual updates they want to roll back to try and resolve issues by themselves. However, Microsoft still controls it for everyone. Where are the instructions for how I can choose what updates to rollback? And also what happens if a security update breaks something?

[Eric_Vernon]

@rocifier Thanks for the feedback. As noted in the blog, this only applies to non-security fixes today. We are exploring solutions for security fixes in the future. For consumers, you are correct there is no choice in the same way that there is no choice when we push updates out for the monthly patch Tuesday. We push the KIR update to mitigate the issue that is affecting consumers. If a consumer is not experiencing the issue, for example the issue is a gaming issue and the individual isn't a gamer, there is no affect to their system unless they start gaming and run into the issue. For individuals experiencing the issue, the configuration change from the cloud solves the issue.

[rocifier]

@Eric_Vernon thanks for replying. The problem I have with that architecture is the following example: Let's say user A (an avid audio listener, but not a gamer) received a Windows update which fixed an issue they were having. Meanwhile user B, who doesn't use surround sound but is an avid gamer receives the same update.

That update had a problem where it means user B cannot play their games in an optimally performant way under common setups. Microsoft decides to push a KIR update which allows the gamer audience, such as user B, a less interrupted experience by rolling back the sound feature from the windows update.

However, in the present process, user A now does not benefit from the released surround sound fix. In an improved architecture, they would have fully benefitted from the fix early while not caring about games being affected at all - and choosing not to receive the rollback. I do understand though that this is perhaps an ideal, but as a regular user of Windows I would like to know that there is at least the option to opt in or out of specific KIR rollbacks, as it suits my usage, though may prefer to receive Microsoft appointed KIR updates if I take no action.

[Nicholas_Steel]

I'd just like KIR updates to, at minimum, notify me before performing their action and give me an opportunity to choose when they are installed... like I've already got set up for regular Windows updates (The WU UI asks if I want to download & install, it doesn't immediately download & install the moment it detects a new update is available).

And for there to be easily found info on what the KIR is going to do.

[giorov92]

Hi, all my games are still having problems since KB5001330.

I tried the suggested group policy fix but that didn't work.

I've restarted my computer several times.

The problem is mostly with game menus and some other fps slowdowns which are not as severe as the menu problems.

[Eric_Vernon]

@Nicholas_Steel @rocifier Thanks for the feedback. As noted, we are evaluating a variety of UI affordances to notify users. At the present time, we do not have that capability. We do post changes on the Windows Health Dashboard (Windows release health | Microsoft Docs) with a pointer to details in the release notes at present.

[mfessler]

Hello everybody,

maybe someone please can help me with the following questions?

- Can the reception resp. the activation of a KIR be initiated/forced?
- How can I find out which KIRs are active on the device?
- Does Microsoft provide an overview of the KIR IDs?
- Does a device that has paused or even deactivated Windows updates still receive possible rollbacks?
- What about systems where telemetry data acquisition/transmission has been deactivated using 3rd party tools for privacy reasons?
Do they still receive KIRs as long as they have not set "AllowExperimentation" to 0, or do receive such systems no KIRs at all?

Thanks and regards,
Martin

[Vatsan_Madhavan]

Thanks for your questions @mfessler.

As we’ve already outlined in the blog, we sometimes share Group Policies intended for Enterprises in the Known Issue Activation KB article associated with a rollback. These Group Policy objects can be used to configure/control rollbacks; that said, at present we don’t have a consumer-centric solution for configuring or enumerating rollbacks.

There is no technical overview of KIR ID’s being made available either; these ID’s are implementation details and aren’t part of the publicly available information. In the future, we may change the implementation approach to use something else besides the ID’s, change the way the ID’s are used and so on – and if we ever make such changes, we’ll ensure that published Group Policies remain in working order. As a general approach we don’t have plans to go into the details of these ID’s at present.  

When Windows Update (WU) is disabled, machines would indeed stop receiving Known Issue Rollbacks. Telemetry is an entirely different lever that only affects whether Windows sends telemetry to Microsoft or not; when a machine is configured to disable telemetry from being sent to Microsoft, then we won’t receive any telemetry related to rollbacks either. That said, such machines would continue receiving (or not receiving) Known Issue Rollbacks depending on whether WU is enabled on those machines (or not).

For the rollbacks to be meaningful, the rollback-metadata must be interpreted in the context of a system that has installed the corresponding (problematic) Update. If a system doesn’t have the corresponding update, the rollback-metadata is benign does nothing further. OTOH if a machine acquires the affected Update (either through WU or some other channel) after the rollback-metadata has been obtained, the rollback will go into effect as soon as the corresponding Update is installed.

The Group Policies we supply with rollbacks also work the same way; machines without the corresponding Update can be pre-configured with the Group Policy if administrators would prefer to do so, and the rollback itself would go into effect as soon as the corresponding Update appears on the systems.

Can you clarify your question about “AllowExperimentation” - what setting is this question about?

Hope this helps!

[Chaython]

KB5001330 KIR was never pushed to me.

Uninstalling KB5001330 via control panel gives an error.

The tutorial on how to manually KIR says it needs a definition MSI, but I can't find one.

How can I fix this? I had already done a repair upgrade to 21h1, but it eventually installed KB5001330 again, with no way to remove it...

[bwprager123]

A remarkable development; but saddening that Microsoft continues its now well-worn pattern of arrogantly taking over every element of the process to prevent it from being easy for less-technically inclined users, particularly home users. It should be pointed out as often as possible that Microsoft's Troubleshooters do very little good; and despite being recommended over and over and over again by MS tech support, they fail to find anything wrong or find that they cannot do the repair. My question which I pray can finally be answered is, will the introduction of this process grant Windows 10 users the restoration of the hated and broken Microsoft Store, and the numerous basic applications which were also destroyed alongside it by Windows Update? Once valuable and basic-beyond-basic applications such as a calculator, alarms and timers, the Photos app and its (hidden) Video Editor are all unavailable and permanently broken, and the hundred-thousand step, highly technical, and nearly-impossible to follow solutions offered in Microsoft's "community" help website require that all non-Microsoft software on the computer must be wiped out in order to have these few basic tools. It's unimaginable that the three non-MS browsers I use, the Zoom program with its carefully assembled meeting schedules, all disc burning tools, all audio file editors, all other photo editors, all PDF viewers and "pro" creator programs, and the dozens of tweaking extensions to mediate the internet, plus all of the word processing, spreadsheet, and other office software, all of the printing devices, all of the particular settings of files into Libraries, the book editors, file organizers and disc management tools must all be destroyed in order to restore less than a handful of very basic tools that are like freeware bits of the Operating System. Sorry for a momentary slip into near-rant mode; but it is simply unconscionable to think that this will continue to be addressed to broad OS issues and fail to restore the applications that every Windows user has already paid for and made part of our daily routines until the notorious Update broke them. I sincerely hope that isn't true; and that isolated corruption of files and individual issue solutions will FINALLY be made available to Windows users. Thanks!!

[Mike_Pisano]

I am going to start this thread with simply “Help”! For me and 1000’s of other engineers struggling with your changes!
I respect you trying to secure you OS for years, but not when you negatively affect business that rely on your product. “File and Print” two main pillars of your OS… Please fix “Point and Print”!!!
I have been supporting you since the 4.77mhz DOS 2.2 days and the birth of Windows!

In case you’re not aware of what you have done… After 250K views, Please review because we are seriously out of options... This is one of many sites!
https://www.bleepingcomputer.com/forums/t/759880/kb5006670-network-printer-problems-again-this-month
One of your last Overrides helped many…

FeatureManagement\Overrides    "713073804"=dword:00000000
We need this extended and your code reverted and whoever is responsible for these changes needs to account for their stupidity.

Please Re-Read and escalate this if you have the ability, I implore you!!!!

[Vatsan_Madhavan]

@Mike_Pisano Thanks for reaching out. 

>  We need this extended and your code reverted and whoever is responsible for these changes needs to account for their stupidity.

I'd like to remind you of our code of conduct - Code of Conduct - Microsoft Tech Community 

https://www.bleepingcomputer.com/forums/t/759880/kb5006670-network-printer-problems-again-this-month is a long thread - is there a specific part of this discussion you'd like to bring to our attention? 

> One of your last Overrides helped many… FeatureManagement\Overrides    "713073804"=dword:00000000

How did you deploy this ? Did you follow the instructions at October 12, 2021—KB5006670 (OS Builds 19041.1288, 19042.1288, and 19043.1288) (microsoft.com) (under Known issues in this update section) and leverage the Group Policies linked there, or did you simply get the Known Issue Rollback from the cloud that directly applied to your PC's ? 

[Mike_Pisano]

@Vatsan_Madhavan 

If the cloud is the thousands of people dealing with this, then Yes; We were going to push it out via GP, but now their is no point if it wont work in the future.....

I had been testing it with 5006670 in lieu of replacing Win32Spl.dll, Localspl.dll and Spoolsv,exe with the September versions (.1237)

Adding 713073804 did fix my issue with 5006670 (.1320) installed, but now with Insiders build KB5007253 (.1379) printing is broken again and this merge no longer works.

Analyzing it in Wireshark, All our issues seem to be related to the newer print stack forcing NTLM_Negotiate  & Challenge  followed by the new encryption on the OpenPrinter().

We need is a permanent way for the Spooler services to function the way it had for decades or else when .1379 (or the final Dec cumulative) rolls out we will be force to roll the 3 files back again or shut down MS updates all together. 

[rocifier]

@Mike_Pisano I may be wrong, but after reading through the forum thread, I only saw it suggested that updating Spoolsv.exe by means of updating windows on the print server could potentially solve the problem, but I don't see that anyone reported actually trying that?

[Mike_Pisano]

@rocifier

Not the Print server, but the clients. (unless you in a RDP environment) and not just SpoolSv but the 3 mentions Bins from an earlier point in time... for us .1237 is stable.

Keep in mind, the .1237 bins still require the new registry entries about approved P&P servers... Copyfiles.... 

Initially I had no other options and no sleep.... 35 sites around the globe still mostly 2003 server with 500 desktops and 500 angry users who cant print. There is still 1 or 2 older Windows 7 machines at each site that everybody used to print out critical documents... (luckily)

I and others have successfully used the older 32\64 bins and the V2 script from the bleeping thread to gain ownership and swap for 2 months now while we wait for MS to "figure this out". We thought the "overrides" might help, and now we learned it's temporary, and expired.

[rocifier]

@Mike_Pisano what I am trying to rule out is whether this is simply an issue of breaking backwards compatibility - if updating the server to match the new client dll can resolve the issue as well, then it may not be a candidate for a KIR.

[jonkloske]

Just to add my 2c worth to Mike Pisano's reports, we run a windows 10 (mix of LTSB, LTSC, and CB - mostly the latter) fleet of approximately 4,000 desktops/laptops with Kerberos authentication/login, and the October update KB5006670 broke 100% of printing for our approximately 10,000 users.

Initial symptoms were that any attempts to print or bring up print properties (or start any app that queries printer lists on startup) would cause the print dialogue or app to just lock up. You could restart the local print spooler service (which would cause any attempted print operations to fail) but would at least un-stick the app. Restarting the server side print spooler would allow a single job through sometimes, but then further attempts would fail with the same "not responding" behaviour.

After a bunch of investigation, we determined that setting this GPO on the print server resolved that part of the issue:

[Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options]

"Network security: Restrict NTLM: Incoming NTLM traffic" = "Deny all accounts"

Unfortunately, whilst this allowed people with existing printers to recommence printing, we quickly discovered that users attempting to add a new printer (specifically, a printer that uses a different driver to the ones they already had installed, or some models of printers like Brother or Canon UFR II drivers) were met with a prompt to elevate to install the driver from the print server, and once you did this you immediately got an error.

The operation failed with error 0x0000007c

We noticed that for most recent Windows 10 Current Branch installs, we could get everything except for the HP printers working by pre-installing the drivers from the print server manually, using twelve variants of the following (one for each printer driver):

pnputil -i -a "\\print.xxxxxxxxxxxxxx\drivers\Konica Minolta\bizhub C558\PS\11.2.0.0\x64\IT5PSWinx64_11200EN\KOAXPA__.INF"
Add-PrinterDriver -Name "KONICA MINOLTA C658SeriesPS"

This did not appear to work for LTSC or LTSB users though. And obviously, doesn't work for HP printers - though the error we got there I think was a slightly different one: in powershell, attempts to add the HP printers threw error 3019 (0xBCB) - ERROR_PRINTER_DRIVER_DOWNLOAD_NEEDED, though I can't remember what the error code was when trying to map it normally. It was different to that.

In any event, we tried GPOs that effectively set the following registry entries:

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Printers\PointAndPrint]

"RestrictDriverInstallationToAdministrators"=dword:00000000

"Restricted"=dword:00000001

"TrustedServers"=dword:00000001

"ServerList"="print.xxxxxxxxxxxxxx;pxxxx.xxxxxxxxxxxx"

"InForest"=dword:00000001

"NoWarningNoElevationOnInstall"=dword:00000001

"UpdatePromptSettings"=dword:00000002

On its own, this only worked for HP plotters and Konica-Minolta printers (the Brother printers actually threw an error about Kernel-mode NT 4.0 drivers being blocked by policy which... is not correct). But combined with the driver preload, allowed 100% of printers to install, and print.

HOWEVER, the printers were only added with the stub printer driver - so all the property sheets and settings usually available from the manufacturer's driver are missing and there's just an extremely limited printer settings dialogue instead. Additionally, some people found this settings dialogue was not correctly creating duplex jobs or sending metadata required for printing like the username etc.

This also doesn't work reliably for LTSC or LTSB, as before - many of these people are stuck with various combinations and hacks of the above.

We've tried the feature known-issue rollback thing on hosts, and have had mixed results with it - it works on some, but not others. And obviously we're missing registry keys for a lot of the versions like LTSB and LTSC, or they don't work. We tried the Get-ScheduledTask -TaskName "ReconcileFeatures" | Start-ScheduledTask  trick as well, and that didn't help. I also noticed the last three octets in the feature disable thing were the same, so I tried all 256 variations on the first octet and also didn't help on these small subset of hosts.

And... that's we are a month and a half later. We're really hoping the December update hinted at that is supposed to help with this does, because if it doesn't, we're looking at rather massive and expensive changes to an extremely large and complex infrastructure just to get printing working.

[Mike_Pisano]

@rocifier 

I wish I had a simple answer...

In my recent experience I can say that my 2016+ servers and Windows 10 clients "both Fully patched" will work as long as all the registry entries (there are many) are pushed to the clients via AD\GP

I will further state that all my Windows 7 client can print regardless since all MS changes seem to be client side in "Accepting" the drivers for "security".

For older\unpatched Windows 10 clients, adding "RpcAuthnLevelPrivacyEnabled = 0" server side seem to allow them to attach to the spoolers.

The last issue is about firewalls, WANS, and authentication protocols failing, especially to older servers like 2003\Xp where RpcAuth does nothing and would be pointless because I don't think it can even handle encryption... We need a client key for "OldSchool" =  1 which would be their "overrides" reg if they could commit to it.

So the KIR is a band-aid and the printer stack needs an UnDO from MS while they try to figure out how do secure it without making it dysfunctional - Which is what they did.....

[Mike_Pisano]

@jonkloske

Thank you for your support and disclosure..... your environment is much larger then mine and if MS can comprehend what you have posted they should really feel sorry for what they did! Countless hours and support calls.

@M$

Please Redmond you are killing us!  Imagine your busiest day, then add 10,000 people calling who can't print!  

Imagine your company can't print your paycheck! Then realize the reality this is the only forum you have left to complain.

[Susan Bradley]

@Vatsan_Madhavan 

are you guys not aware of this post on bleeping computer and corresponding forum threads?

How to fix the Windows 0x0000007c network printing error (bleepingcomputer.com)

I think Mike might be thinking of this post where the Oldgrumpyadmin poster is pointing to a workaround that was given to him by someone in Microsoft support -- KB5006670 - Network Printer Problems Again This Month - Page 19 - Windows 10 Support (bleepingcomput... 

I've also seen people replace the spool file with an older one to get things Topic: One Man’s Tale of Update Hell @ AskWoody

[Susan Bradley]

P.S folks that's 21 PAGES of forum posts regarding this problem.  Microsoft did an out of band for the kerberos/single sign on.  How about an out of band or some better accumulation of guidance on this?  Replacing patched dlls isn't wise but that's what people are doing to get printing to work.  Even with the pandemic, we still have to print.

Back a few months ago I had to rip out all of my V3 printer drivers and replace them all with v4 to get printing working the first time the print spooler fixes started breaking things.  But I'm not having to manage 10,000 users.