Shims – Are They Really Such Nasty Bits?

[Chris Jackson]

First posted to MSDN on Oct, 20 2010

I wanted to post a response to what I consider to be somewhat suboptimal advice, with the hopes of correcting it. In recorded history, there have been a number of classic blunders when it comes to advice. They include: “ Hey, let’s start a land war in Asia! ” “ I really think you should buy the Betamax instead of the VHS. ” “ Let’s go a little faster. Nothing can sink this ship – it’s the freaking Titanic! ” And now, to add to that list, is my friend Greg Lambert’s latest advice on shims: http://aokcompat.blogspot.com/2010/10/shims-those-nasty-bits.html Before I get into a discussion on this, let me make sure that my position is absolutely clear: nobody looks forward to using shims. It’s not something that I recommend that anybody go out of their way to use, or keep around any longer than absolutely necessary. It’s not something that you have to use in order to be successful, nor is it necessarily your first choice. But, if you want to manage costs of your deployment (who doesn’t?) then I strongly recommend keeping this option in your toolbox. As you troubleshoot compatibility problems, you’re going to build a list of technical options for what can be done when something doesn’t work. You can, of course, always buy or build a new one, but that’s the most expensive choice. (It’s also the best choice in the long run, if you’re going to be keeping the app around for a significant length of time – but it’s often nice to divorce the timelines of this option from the timeline of your deployment.) You can also virtualize and continue running it on a previous operating system, but then you have to manage that operating system and buy machines with enough resources to run it (also not free). After that, you have a list of things which may work. Maybe you can shim it, maybe not. Maybe you can modify the MSI, maybe not. None of them is a catch-all, so if you remove an option, you’re going to be able to fix less, which means you’re going to be forced to select one of the more expensive options. Unless your management gave you a budget of $∞.00 you may not want to exclude a viable option. In my experience, people include this option technically, and then build business rules to include it from a business perspective in cases such as:

• The update is going to take too long, so I just want to buy some time
• The code with the bug was replicated across a number of our apps – rather than waiting to fix them all, I can deploy today and have the shims roll off as they gradually upgrade these apps to fix all occurrences of this bug
• The company is out of business, so we need time to build or buy an alternative – it’s not clear what that is today, and I have no other way to fix it in a cost-effective way

I do recommend thinking through and determining the scenarios where it’s OK for your project team to use shims, because they are definitely a duct-tape solution. So, with that bit of an intro, let’s examine Greg’s points: Microsoft owns the Shim database and may update the database at anytime I admit, I don’t understand this argument. The fact that we use shims to resolve compatibility issues doesn’t seem like the most compelling issue for you not to use it. In fact, since this what we primarily choose to fix app compat issues, it seems like a fairly strong endorsement for it. Keep in mind, we don’t own all shim databases – we own ours. sysmain.sdb contains the commercial software we’re fixing, and of course we’d like to be able to fix more applications that we weren’t aware of when we shipped Windows 7, because people like their software to work whether or not we happened to have a copy or a bug report prior to July 22, 2009. If you make your own (one or more), it’s yours, and we don’t touch that. We’re not going to overwrite your work. No published API’s for the Shim Database This argument is not exactly true. There are published APIs for the shim engine. They are here: http://msdn.microsoft.com/en-us/library/bb432182(v=VS.85).aspx . However, it would be fair to say that there are no easy-to-use APIs for managing shim databases. This affects Greg from the perspective of building a tool to automate fixes, making his job unnecessarily difficult, and it’s something that we’re actively investigating to make his job as a partner with Microsoft easier. However, it’s not something that affects the enterprise – I am not aware of any blocked scenarios in the enterprise as a result of the lack of easy-to-use C APIs for the shim engine. So, I don’t consider this a compelling argument for enterprise adoption, though it’s still an action item I’d like to do a better job with. No published documentation for the Shim Database schema Same comment as the above for APIs – it doesn’t affect the enterprise. We also don’t publish all of the internal structures and code used in structured exception handling, and we still recommend that you use that. I believe this is also a result of conflating the needs of the ISV from the needs of the enterprise. No tools for managing the Shim Database in an enterprise environment While our tooling could certainly be improved, we provide tools for creating (Compatibility Administrator) and installing / updating / removing (sdbinst.exe) shim databases. We are very interested in hearing about the scenarios where better tooling is required. But, since this is a duct tape solution, it’s something we hope you aren’t actively managing day to day, but rather something you put in place initially and hopefully never have to touch again, or at least only have to touch once or twice more. If you’re routinely fiddling with shims, that’s admittedly a bit unexpected. We are very interested in hearing expectations and experiences as adoption of this mitigation grows. No Active Directory integration There are many things which are not AD integrated, yet are still a good idea. Pecan Pie is not integrated with AD, yet Pecan Pie is freaking awesome. Personally, I’d like to see a GPO-based solution, but it’s not completely clear how that would work, or how it would work better than what you would do today, which is through pushing a script. So, this seems like the same argument as the last one: management exists, but could certainly be better. I agree, but don’t consider that an insurmountable obstacle. IT Administrators do many, many things with scripts today where GPOs don’t cover specific scenarios. No native integration with 3rd part deployment tools Same as above – any tool which supports scripting can be used, and management requires a single line of script. Any explanation of the specific requirements of the enterprise can help us to focus investments. We don’t want to over-engineer what we hope to be a very temporary solution, but we want to do enough. No central management and control of the Shim Database Same as above. No centralized reporting mechanism’s available (from MS or ISV’s) Same as above. We do have instrumentation, as documented here: http://blogs.msdn.com/b/cjacks/archive/2008/05/20/enabling-diagnostic-output-from-shims.aspx Any update to the Shim Database requires a complete re-compile of the database I’m not sure why this is a problem, per-se. First, I’m curious as to how often people are updating them – for my customers, it’s not terribly often. Second, since a recompile is so fast and can be done with any computer, and starts off with the ability to reverse any existing database (installed or not) so source code maintenance is not required, I don’t understand why this is a blocker. In fact, I don’t understand the scenario (outside of the ISV scenario) where you would want otherwise? No transparency or “accounting” of changes or updates to a Shim database Shim databases are completely transparent – you can view them on any computer using the Compatibility Administrator tool. Now, when you update to a new version, there is no reporting on the delta, so you’d have to manually compare, but it can be done. But my opinion here is that it’s relatively transparent, of little interest to the end user, and the administrator who deploys the update should be completely aware of what changed, since they, after all, were the ones who changed it. Perhaps I don’t understand the scenario well enough? No “back-out” or roll-back facilities for the Shim Database You can uninstall a database with sdbinst /u very easily. You can also roll back to a previous version by calling sdbinst and pointing at the previous version (SDBInst will overwrite any custom shim database with the same GUID, whether moving backwards or forwards). Limited Multi-Platform Support I would go as far as to say zero multi-platform support. While you can create an SDB on a different OS than the one you deploy it to, it’s generally a bad idea to do so. While the shims you apply may exist, the semantics might change. And, if they don’t exist, it won’t wire up anything. But I think this indicates a misunderstanding of the intent of shims. They are not designed to be portable. The shim database is an extension to a particular OS to make the applications multi-platform by fixing the bugs that stop them from doing so, and shouldn’t be carried around between OSs. An app may run natively on Windows XP, need one set of shims to run on Windows Vista, and a separate set of shims to run on Windows 7. You don’t package the shims with the app and expect them to work everywhere, you package the shims with the OS so the apps that you want work on the OS that you want. We seem to be associating the shim with the wrong entity – they belong with the OS, not with the app. To support these points, Greg then quotes me:

It is possible that you will eventually find that the shims assigned to resolve a set of compatibility issues in an application are not comprehensive and that later you will need to deploy an updated version of the custom shim database that resolves the additional issues your organization later discovered. If you deployed the original custom shim database as part of the installation package, you will need to locate each client that has installed this application and the original custom shim database for it to replace it with the new version.

It’s fairly important to take this quote in context, so let’s apply the context and clarify this quote. There are a couple of primary choices people look at for deploying shims: deploy them with the app, or deploy them with the OS. In most all situations, I personally believe that the better choice is to deploy them with the OS, and manage a single central shim database rather than a number of stand-alone shim databases containing a single app entry which you deploy with each installer. While that method is OK when you only deploy one or two and in general seek to avoid them, as soon as it goes even slightly behind a one-off, I believe it makes sense to have a single, centralized shim database for the organization based on the input from many customers I have worked with. So, it’s somewhat disingenuous to take a quote in an argument against choosing a particular methodology of deployment and generalize it as an argument against using the technology. That would be like declaring that deploying Windows 7 is a bad idea because deploying it via sneakernet (having a guy walk around with a DVD and manually install it) has its disadvantages for a multinational corporation.

Greg then points out his recommendation, which is to use “Best Practices offered by Microsoft”: Elevation Manifests It is emphatically not a best practice to apply an elevation manifest to somebody else’s code. Under no circumstances do I believe this is a good idea. In suppressing or instigating an elevation prompt, the very act of adding a UAC manifest disables many compatibility features in the operating system. For example, you lose file and registry virtualization. That means you have to manually remediate many of the most common file and registry permissions issues. Since the argument is to not use shims, then you are left with the old Windows XP style of remediating these applications: by modifying permissions. There is a reason why not many customers migrated to standard users on Windows XP: because migrating without these assistive technologies was too difficult and expensive. Please, do not apply a manifest to code you have not written. This is an anti-best-practice in the application compatibility project scenario, and will potentially lower security and definitely generate more work. Apply elevation flags via a shim database instead. Updating the source application package (MSI Installers) Absolutely! Depending on the customer, statistically the average number of application issues that are installer issues can be up to 40%, sometimes even more, so you have to fix things here in the package. There are also a number of issues where you can drop working code and remove the buggy code, which is definitely preferable to shimming buggy code when that is an option. I enthusiastically agree, but believe that this is orthogonal. There are many things you can’t fix with shims which you can fix by changing the package, and there are many things you can’t fix in the package which you can fix with shims. It’s good to have both tools on the approved list, and if both are an option, I’ll almost always choose this one! Greg is spot on here. Creating Side-by-Side (Sxs) Isolation manifests

In my experience, this resolves a relatively small number of potential issues, so I seldom find this one drifting to the top of my list. However, it is another potential tool which I keep in the back of my mind when it comes up as appropriate, and my anecdotal experiences around frequency aren’t necessarily representative.

In summary, while I don’t necessarily advocate the opposite of Greg’s advice and wouldn’t say that you absolute must use shims. Instead, I recommend that you don’t rule them out straight away. Rather, I’d consider them a valid technical option, and then make the business decision for when you choose to utilize this, among other, technical options. Doing so can help you to minimize cost and accelerate your deployments. They’re simply too valuable in too many situations for you to avoid them outright because of fear, uncertainty, or doubt.