The trouble with "bitness" (aka architecture) and shiming.

%3CLINGO-SUB%20id%3D%22lingo-sub-239921%22%20slang%3D%22en-US%22%3EThe%20trouble%20with%20%22bitness%22%20(aka%20architecture)%20and%20shiming.%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-239921%22%20slang%3D%22en-US%22%3E%3CP%3EI%20want%20to%20highlight%20an%20issue%20with%20implementing%20PSF%20shimming%20that%20I%20ran%20into.%26nbsp%3B%20It%20was%20particularly%20annoying%20because%20the%20symptoms%20did%20not%20match%20the%20real%20problem.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20issue%20is%20the%20need%20for%20alignment%20of%20the%20shim%20launcher%2C%20the%20shims%2C%20and%20the%20target%20application%20when%20it%20comes%20to%20what%20I%20call%20%22bitness%22.%26nbsp%3B%20%22Bitness%22%2C%20is%20how%20I%20describe%20if%20the%20app%2C%20or%20a%20component%2C%20is%2032-bit%20or%2064-bit.%26nbsp%3B%20This%20shows%20up%20in%20some%20tooling%20as%20%22Architecture%22%2C%20but%20as%20an%20overused%20term%20that%20one%20can%20be%20confusing%20to%20people.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHere%20is%20what%20I%20(think%20I)%20know%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EStarting%20out%20with%20the%20easy%20part%2C%20if%20the%20target%20application%20is%20a%2032-bit%20exe%20then%20you%20need%20to%20use%20the%2032-bit%20launcher%20and%2032-bit%20shims.%26nbsp%3B%20If%20the%20target%20application%20is%2064-bit%2C%20then%20you%20need%20to%20use%20the%2064-bit%20launcher%20and%20shims.%26nbsp%3B%20The%20impact%20of%20this%20is%2C%20unless%20we%20build%20a%20radically%20different%20launcher%20than%20in%20the%20PSF%20repository%20currently%2C%20it%20will%20become%20necessary%20to%20have%20independent%2032%20and%2064%20bit%20MSIX%20packages%20if%20the%20vendor%20has%2032%20and%2064%20bit%20exes%20for%20the%20application.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EBut%20then%20it%20gets%20messy%20with%20%22AnyCPU%22%20executables.%26nbsp%3B%20AnyCPU%20is%20a%20build%20option%20that%20produces%20a%20single%20exe%20or%20dll%20that%20will%20run%2032-bit%20on%20a%2032-bit%20OS%20and%2064-bit%20on%20x64%20(I%20believe%20that%20on%20x64%20an%20AnyCPU%20dll%20will%20run%20in%20the%20bitness%20of%20the%20process%20that%20loaded%20it).%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EOn%20an%20x64%20OS%20when%20a%2032-bit%20launcher%20creates%20the%20application%20process%20using%20an%20AnyCPU%20exe%20the%20application%20process%20becomes%2064-bit%20(configurable%20in%20the%20linking%2C%20but%20by%20default%2064)%20and%20will%20need%2064-bit%20shims.%26nbsp%3B%20So%20even%20though%20the%20vendor%20supplies%20only%20one%20exe%2C%20we%20still%20need%20two%20different%20packages%20with%20different%20bitness%20shims%20(and%20probably%20launcher%20for%20simplicity).%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EBut%20it%20is%20NOT%20obvious%20that%20an%20executable%20is%20AnyCPU.%26nbsp%3B%20And%20as%20I%20learned%20over%20the%20past%20two%20days%20not%20so%20easy%20to%20diagnose%20as%20CreateProcess%20returns%20with%20a%20file%20not%20found%20error.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIf%20PSF%20is%20going%20to%20be%20%22easy%22%2C%20we%20need%20to%20make%20bitness%20issues%20handled%20automatically.%26nbsp%3B%20As%20the%20current%20shims%20are%20C%2B%2B%20based%20I%20don't%20think%20we%20can%20make%20them%20AnyCPU.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-324694%22%20slang%3D%22en-US%22%3ERe%3A%20The%20trouble%20with%20%22bitness%22%20(aka%20architecture)%20and%20shiming.%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-324694%22%20slang%3D%22en-US%22%3E%3CP%3EYes%20-%20%22bitness%22%20describes%20it%20quite%20well.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20you%20are%20building%20an%20installer%2C%20you%20can%20include%20all%20components.%26nbsp%3B%20But%20if%20you%20are%20an%20IT%20pro%20repackaging%20with%20%22the%20tool%22%2C%20the%20installer%20only%20puts%20down%20the%20bitness%20specific%20bits%20associated%20with%20the%20OS%20you%20capture%20on.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EGiven%20that%20we%20are%20capturing%20and%20deploying%20to%20Windows%2010%20-%20this%20isn't%20much%20of%20an%20issue%20because%20nobody%20uses%20the%2032-bit%20version%20of%20the%20OS%20(well%2C%20except%20in%20VMs%20where%20memory%20matters%20I%20suppose).%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIt%20sounds%20like%20the%20PSF%20components%20actually%20handle%20more%20of%20this%20automatically%20than%20I%20thought%20-%20making%20sure%20they%20insert%20the%20right%20bitness%20shims%20if%20we%20supply%20them.%20But%20until%20I%20can%20test%20real-world%20scenarios%20on%20this%20I'll%20hold%20on%20making%20a%20final%20judgement.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-324684%22%20slang%3D%22en-US%22%3ERe%3A%20The%20trouble%20with%20%22bitness%22%20(aka%20architecture)%20and%20shiming.%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-324684%22%20slang%3D%22en-US%22%3E%3CP%3EGood%20points.%20However%2C%20all%20commercial%20apps%20that%20I've%20seen%20where%20the%20releases%20yielded%20x86%2C%20x64%2C%20and%20AnyCPU%20build%20did%20just%20that%20-%20made%20all%20three%20platforms%20available%20(or%20bitness%20-%20I%20like%20that%20word%20too.)%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20AnyCPU%20is%20problematic%20with%20the%20PSF%20shim%20launcher%2C%20why%20not%20package%20with%20the%20x86%20or%20x64%20build%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1158252%22%20slang%3D%22en-US%22%3ERe%3A%20The%20trouble%20with%20%22bitness%22%20(aka%20architecture)%20and%20shiming.%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1158252%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F146612%22%20target%3D%22_blank%22%3E%40TIMOTHY%20MANGAN%3C%2FA%3E%26nbsp%3Byou%20can%20use%20corflags.exe%20from%20the%20Windows%20SDK%20to%20determine%20if%20an%20app%20is%20AnyCPU%20btw%3A%3CBR%20%2F%3E%3CBR%20%2F%3E%3CA%20href%3D%22https%3A%2F%2Fstackoverflow.com%2Fquestions%2F18608785%2Fhow-to-interpret-the-corflags-flags%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fstackoverflow.com%2Fquestions%2F18608785%2Fhow-to-interpret-the-corflags-flags%3C%2FA%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%3CPRE%3ECPU%20Architecture%20%20%20%20%20%20%20%20%20%20%20PE%20%20%20%20%20%2032BITREQ%20%20%2032BITPREF%0A------------------------%20%20%20-----%20%20%20--------%20%20%20---------%0Ax86%20(32-bit)%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20PE32%20%20%20%20%20%20%20%20%20%20%201%20%20%20%20%20%20%20%20%20%20%200%0Ax64%20(64-bit)%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20PE32%2B%20%20%20%20%20%20%20%20%20%200%20%20%20%20%20%20%20%20%20%20%200%0AAny%20CPU%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20PE32%20%20%20%20%20%20%20%20%20%20%200%20%20%20%20%20%20%20%20%20%20%200%0AAny%20CPU%2032-Bit%20Preferred%20%20%20PE32%20%20%20%20%20%20%20%20%20%20%200%20%20%20%20%20%20%20%20%20%20%201%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1158917%22%20slang%3D%22en-US%22%3ERe%3A%20The%20trouble%20with%20%22bitness%22%20(aka%20architecture)%20and%20shiming.%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1158917%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F145709%22%20target%3D%22_blank%22%3E%40Dan%20Gough%3C%2FA%3E%20Well%2C%20I%20do%20that%20in%20PsfTooling%20to%20know%20which%20components%20to%20install%2C%20but%20it%20gets%20more%20complicated%20too.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20AnyCPU%2C%20the%20dev%20can%20specify%20what%20the%20default%20should%20be%20-%20this%20affects%20how%20the%20AnyCPU%20will%20run%20if%20on%20an%20x64%20machine.%26nbsp%3B%20Visual%20Studio%20used%20to%20default%20this%20default%20setting%20x64%2C%20but%20now%20defaults%20to%20x86.%20But%20then%20it%20isn't%20just%20the%20file%20because%20we%20also%20have%20a%20GPO%20for%20the%20target%20system%20that%20can%20override%20that%20default!%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EUltimately%20I%20resolved%20the%20PsfMonitor%20issue%20by%20changing%20away%20from%20AnyCPU%20and%20the%20PSF%20now%20builds%20bitness%20specific%20versions%20of%20that%20component%20too.%20%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1166500%22%20slang%3D%22en-US%22%3ERe%3A%20The%20trouble%20with%20%22bitness%22%20(aka%20architecture)%20and%20shiming.%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1166500%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F146612%22%20target%3D%22_blank%22%3E%40TIMOTHY%20MANGAN%3C%2FA%3E%26nbsp%3BThe%20PSF%20%3CSTRONG%3ELauncher%3C%2FSTRONG%3E%26nbsp%3Bexecutable%20comes%20in%20both%2032%20and%2064%20bit%20variants%2C%20both%26nbsp%3B%3CSPAN%3Eof%20them%20being%20able%20to%20launch%20any%20kind%20of%20process%20as%20long%20as%20the%20PSF%20%3CSTRONG%3ERuntime%26nbsp%3B%3C%2FSTRONG%3Edll%20with%20the%20same%20bitness%20as%26nbsp%3Bthe%20launched%20process%20is%20also%20present%20in%20the%20package%20with%20a%20corresponding%20name%20(PsfRuntime.dll%20or%20PsfRuntime%3CEM%3E%5Barch%5D%3C%2FEM%3E.dll).%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3EAnother%20PSF%20component%20available%20is%20%3CSTRONG%3EPsfRunDll%3C%2FSTRONG%3E%20executable%20which%20is%20needed%26nbsp%3Bonly%20if%20the%20main%20application%20process%20creates%20additional%20processes%20(with%20different%20bitness)%2C%20so%20I%20would%20say%20that%20the%26nbsp%3B%3CSPAN%3Ebitness%20issues%20%3CSTRONG%3Eare%20being%20handled%20automatically%3C%2FSTRONG%3E%20by%20the%20PSF.%3C%2FSPAN%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22inherit%22%3EIf%20you%20are%20using%20a%20tool%20with%20PSF%20support%2C%20like%26nbsp%3BAdvanced%20Installer%2C%20all%20that%20bootstrapping%20is%20being%20done%20behind%20the%20scenes%2C%20you%20just%20have%20to%20pick%20the%20predefined%20fixup%20you%20need%20since%20all%20are%20%3C%2FFONT%3Eavailable%3CFONT%20face%3D%22inherit%22%3E%26nbsp%3Bfor%20both%20architectures.%3C%2FFONT%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
MVP

I want to highlight an issue with implementing PSF shimming that I ran into.  It was particularly annoying because the symptoms did not match the real problem.

 

The issue is the need for alignment of the shim launcher, the shims, and the target application when it comes to what I call "bitness".  "Bitness", is how I describe if the app, or a component, is 32-bit or 64-bit.  This shows up in some tooling as "Architecture", but as an overused term that one can be confusing to people.

 

Here is what I (think I) know:

 

Starting out with the easy part, if the target application is a 32-bit exe then you need to use the 32-bit launcher and 32-bit shims.  If the target application is 64-bit, then you need to use the 64-bit launcher and shims.  The impact of this is, unless we build a radically different launcher than in the PSF repository currently, it will become necessary to have independent 32 and 64 bit MSIX packages if the vendor has 32 and 64 bit exes for the application.

 

But then it gets messy with "AnyCPU" executables.  AnyCPU is a build option that produces a single exe or dll that will run 32-bit on a 32-bit OS and 64-bit on x64 (I believe that on x64 an AnyCPU dll will run in the bitness of the process that loaded it). 

 

On an x64 OS when a 32-bit launcher creates the application process using an AnyCPU exe the application process becomes 64-bit (configurable in the linking, but by default 64) and will need 64-bit shims.  So even though the vendor supplies only one exe, we still need two different packages with different bitness shims (and probably launcher for simplicity). 

 

But it is NOT obvious that an executable is AnyCPU.  And as I learned over the past two days not so easy to diagnose as CreateProcess returns with a file not found error. 

 

If PSF is going to be "easy", we need to make bitness issues handled automatically.  As the current shims are C++ based I don't think we can make them AnyCPU.

5 Replies
Highlighted

Good points. However, all commercial apps that I've seen where the releases yielded x86, x64, and AnyCPU build did just that - made all three platforms available (or bitness - I like that word too.) 

 

If AnyCPU is problematic with the PSF shim launcher, why not package with the x86 or x64 build?

Highlighted

Yes - "bitness" describes it quite well.

 

If you are building an installer, you can include all components.  But if you are an IT pro repackaging with "the tool", the installer only puts down the bitness specific bits associated with the OS you capture on.

 

Given that we are capturing and deploying to Windows 10 - this isn't much of an issue because nobody uses the 32-bit version of the OS (well, except in VMs where memory matters I suppose).

 

It sounds like the PSF components actually handle more of this automatically than I thought - making sure they insert the right bitness shims if we supply them. But until I can test real-world scenarios on this I'll hold on making a final judgement.

Highlighted

@TIMOTHY MANGAN you can use corflags.exe from the Windows SDK to determine if an app is AnyCPU btw:

https://stackoverflow.com/questions/18608785/how-to-interpret-the-corflags-flags

CPU Architecture           PE      32BITREQ   32BITPREF
------------------------   -----   --------   ---------
x86 (32-bit)               PE32           1           0
x64 (64-bit)               PE32+          0           0
Any CPU                    PE32           0           0
Any CPU 32-Bit Preferred   PE32           0           1

 

Highlighted

@Dan Gough Well, I do that in PsfTooling to know which components to install, but it gets more complicated too.

 

In AnyCPU, the dev can specify what the default should be - this affects how the AnyCPU will run if on an x64 machine.  Visual Studio used to default this default setting x64, but now defaults to x86. But then it isn't just the file because we also have a GPO for the target system that can override that default!

 

Ultimately I resolved the PsfMonitor issue by changing away from AnyCPU and the PSF now builds bitness specific versions of that component too.  

Highlighted

@TIMOTHY MANGAN The PSF Launcher executable comes in both 32 and 64 bit variants, both of them being able to launch any kind of process as long as the PSF Runtime dll with the same bitness as the launched process is also present in the package with a corresponding name (PsfRuntime.dll or PsfRuntime[arch].dll).

 

Another PSF component available is PsfRunDll executable which is needed only if the main application process creates additional processes (with different bitness), so I would say that the bitness issues are being handled automatically by the PSF.

 

If you are using a tool with PSF support, like Advanced Installer, all that bootstrapping is being done behind the scenes, you just have to pick the predefined fixup you need since all are available for both architectures.