Investigating Windows Vista Compatibility Issues by Decompiling VB Code: Don't Be Afraid to Cheat!
Published Aug 14 2018 03:57 PM 172 Views
Microsoft
First posted to MSDN on Dec, 07 2007

So, the other day I was investigating a visual issue in an application. On Windows 2000, the background of the form was white. On Windows Vista, in Windows Classic mode, the background of the form was white. But, for whatever reason, in either Windows Vista Aero or Windows Vista Basic, the background of the form was black. And it was a little hard to read that way.

The first thing I did was try to find something out about the application, but in this case is was an in-house application. The best I could do was to poke around the application and gather some facts so I could form my hypothesis on what could be going wrong, and how I could fix it.

A quick glance at the output of lm in the debugger revealed that msvbvm50.dll was loaded. OK, so we have a Windows 2000 application built on Visual Basic 5.0. This strongly suggested that the application wasn't leveraging visual styles and the functions in uxtheme.dll to pick up colors (which also meant that the FakeLunaTheme shim probably wouldn't do any good, and in fact it didn't - I tried anyway).

So then I started going down the road of setting breakpoints on functions such as GetSysColor to see what is going on, and then I was attempting to mentally translate the code from ASM to VB5. Oh, and I was doing this with public symbols too, which made it even more fun.

Then, I got a clever idea. What if I didn't have to do the translation in my head? I mean, Lutz Roeder's awesome Reflector for .NET is unbelievably useful for prying secrets from managed code, I wondered if anybody had ever done the same for VB5/6 code? It seemed like it could be possible...?

Some quick web searching later, and I came up with VBReFormer . Ah - and they have a free edition! Let's give that a go.

Literally 2 seconds later, I was at the form, looking at properties, and the answer was right in front of my face. The developer has set the form background color to be the system color for Title Bar Text. (Seriously.) And, alas, we don't have a shim for that one. But we could just change the title bar text, the easiest way being to switch to Windows Classic mode when using this app. Or, just put up with the fact that the black background didn't look as nice (at least the text was blue, so it wasn't unreadable as it would have been if it were black text).

And this is yet another example of what I am always preaching. There is a certain geek pride in going after a deep understanding of what's going on. But there are no prizes for spending the most time in a debugger. In fact, it's a great way to make time vanish if you aren't careful. Use the tool to get the answer to your question - prove or disprove a hypothesis - and then get out. If there is an easier tool, use it. If you think "wow, I bet this tool would be awesome if it existed" then poke around - it very well may.

Most of all, don't be afraid to cheat. Nobody is counting the number of times you typed bp in a debugger, or how quickly you can ferret out the calling convention to sort out arguments with public symbols. But they do tend to notice when you fix things.

Oh, and kudos to the VBReFormer team for saving me a boatload of time. I don't know if there is a better tool, I'm not an expert in that space and this is the first and only one I looked at. But I was thrilled that I didn't have to hand over that portion of my life to a debugger.

Version history
Last update:
‎Nov 13 2018 08:16 AM
Updated by: