CSI: Redmond - Episode 1 “Mistaken Identity”

Published 04-19-2021 04:57 PM 1,429 Views
Microsoft

 

 

Episode Story Line: 

 

A mild mannered IT administrator is doing routine patching when things go south.  At first it seems like maybe just a case of the wrong patch for the wrong system but there is more to this story.   Time to put our detective skills to use and find out what’s going on.

 

image001.png

 

The Investigation:

 

Any patch installation investigation failure should start with the basics, aka the Application/System/Security event logs.  In this case, the normal logs simply re-state the error message displayed on screen so no help there…it looks like we’re already past the basics.

 

EventLogPstchFail1.png

 

If you have ever performed troubleshooting on in-depth patch installation, the CBS log (c:\Windows\Logs\CBS\CBS.log) is your go to source and contains a wealth of information on the installation process, some would say too much information.  But that’s why we’re here today, to help find the clues to solve the crime.

 

In this case, if you go directly to the end of the CBS log after attempting to install the patch, you should see the tail end of the process.  We don’t have to look too far before seeing signs of a problem:  ERROR_NOT_SAME_DEVICE… what’s that all about? \

 

image003.png

 

One trick I use in sifting through the large amount of data in the CBS.log is to search for key words, like “ERROR” or “FAIL”, but in the clues we are searching for can quickly be found by searching for the term “CSI” (see where the title of this post comes from?).  CSI stands for Component Servicing Infrastructure and is responsible for putting the patch files on the system.  If you are getting errors due to permissions or possibly from antivirus products, this is the category you want to look for. Don’t forget to search in the “Up” direction…gets me every time. Note the line below found while searching the CSI messages.  The reference to Volume serial number as well as the name of the target file, which references a group policy administrative template, were instrumental fingerprints in this case.

 

image004.png 

 

Now, let’s go take a look at that location to see if anything looks wonky.  The first thing we notice is that there is a PolicyDefinitions_old folder, which appears to be a backup of the unchanged folder…at least our suspect is cautious.  Notice the “Shortcut” icon on the new PolicyDefinitions folder.  Things are starting to heat up…getting warmer.

 

SymLink.png

 

To get more information on the folder, let’s go to PowerShell and take a look at the object with Get-Item C:\Windows\PolicyDefinitions | fl *.  While the FullName field looks correct, the Target field references another volume entirely.  This is what is causing the patch installation to fail.  After talking to the server owner, it came to light that a GPO management tool was installed on the system and the link was created to assist the product manage the admin templates.  The actual software had been installed on a different volume, hence the Junction point going to a different drive.

 

Symlink_DirJunctionPoSh.png

 

The Fix:

 

The fix for this situation was to remove the Junction point and re-create the PolicyDefinitions folder so that the patch could install the new templates.  Since the Junction was a requirement, we had to copy the new admin templates to the target folder and then restore that junction point temporarily but are actively working to come up with a solution that does not trigger the different volume problem.

Another tough issue put to bed but I’m sure many more will follow.            

%3CLINGO-SUB%20id%3D%22lingo-sub-2278566%22%20slang%3D%22en-US%22%3ECSI%3A%20Redmond%20-%20Episode%201%20%E2%80%9CMistaken%20Identity%E2%80%9D%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2278566%22%20slang%3D%22en-US%22%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CDIV%20class%3D%22WordSection1%22%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%3CFONT%20color%3D%22%230000FF%22%3E%3CSTRONG%3E%3CSPAN%20class%3D%22Heading1Char%22%3E%3CSPAN%20style%3D%22font-size%3A%2016.0pt%3B%20line-height%3A%20107%25%3B%22%3EEpisode%20Story%20Line%3A%3C%2FSPAN%3E%3C%2FSPAN%3E%26nbsp%3B%3C%2FSTRONG%3E%3C%2FFONT%3E%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3EA%20mild%20mannered%20IT%20administrator%20is%20doing%20routine%20patching%20when%20things%20go%20south.%26nbsp%3B%20At%20first%20it%20seems%20like%20maybe%20just%20a%20case%20of%20the%20wrong%20patch%20for%20the%20wrong%20system%20but%20there%20is%20more%20to%20this%20story.%26nbsp%3B%20%26nbsp%3BTime%20to%20put%20our%20detective%20skills%20to%20use%20and%20find%20out%20what%E2%80%99s%20going%20on.%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22image001.png%22%20style%3D%22width%3A%20597px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F273922i9D76C45AB3349A82%2Fimage-dimensions%2F597x424%3Fv%3Dv2%22%20width%3D%22597%22%20height%3D%22424%22%20role%3D%22button%22%20title%3D%22image001.png%22%20alt%3D%22image001.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId-1304012955%22%20id%3D%22toc-hId-1303325779%22%3E%3CFONT%20color%3D%22%230000FF%22%3EThe%20Investigation%3A%3C%2FFONT%3E%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3EAny%20patch%20installation%20investigation%20failure%20should%20start%20with%20the%20basics%2C%20aka%20the%20Application%2FSystem%2FSecurity%20event%20logs.%26nbsp%3B%20In%20this%20case%2C%20the%20normal%20logs%20simply%20re-state%20the%20error%20message%20displayed%20on%20screen%20so%20no%20help%20there%E2%80%A6it%20looks%20like%20we%E2%80%99re%20already%20past%20the%20basics.%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22EventLogPstchFail1.png%22%20style%3D%22width%3A%20788px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F273938i81A878C51C02E990%2Fimage-dimensions%2F788x661%3Fv%3Dv2%22%20width%3D%22788%22%20height%3D%22661%22%20role%3D%22button%22%20title%3D%22EventLogPstchFail1.png%22%20alt%3D%22EventLogPstchFail1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3EIf%20you%20have%20ever%20performed%20troubleshooting%20on%20in-depth%20patch%20installation%2C%20the%20CBS%20log%20(%3CSTRONG%3E%3CI%3Ec%3A%5CWindows%5CLogs%5CCBS%5CCBS.log%3C%2FI%3E%3C%2FSTRONG%3E)%20is%20your%20go%20to%20source%20and%20contains%20a%20wealth%20of%20information%20on%20the%20installation%20process%2C%20some%20would%20say%20too%20much%20information.%26nbsp%3B%20But%20that%E2%80%99s%20why%20we%E2%80%99re%20here%20today%2C%20to%20help%20find%20the%20clues%20to%20solve%20the%20crime.%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3EIn%20this%20case%2C%20if%20you%20go%20directly%20to%20the%20end%20of%20the%20CBS%20log%20after%20attempting%20to%20install%20the%20patch%2C%20you%20should%20see%20the%20tail%20end%20of%20the%20process.%26nbsp%3B%20We%20don%E2%80%99t%20have%20to%20look%20too%20far%20before%20seeing%20signs%20of%20a%20problem%3A%26nbsp%3B%20%3CSTRONG%3E%3CI%3EERROR_NOT_SAME_DEVICE%3C%2FI%3E%3C%2FSTRONG%3E%E2%80%A6%20what%E2%80%99s%20that%20all%20about%3F%20%5C%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22image003.png%22%20style%3D%22width%3A%20798px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F273948iD2413AD3A7A2511E%2Fimage-dimensions%2F798x423%3Fv%3Dv2%22%20width%3D%22798%22%20height%3D%22423%22%20role%3D%22button%22%20title%3D%22image003.png%22%20alt%3D%22image003.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3EOne%20trick%20I%20use%20in%20sifting%20through%20the%20large%20amount%20of%20data%20in%20the%20CBS.log%20is%20to%20search%20for%20key%20words%2C%20like%20%E2%80%9CERROR%E2%80%9D%20or%20%E2%80%9CFAIL%E2%80%9D%2C%20but%20in%20the%20clues%20we%20are%20searching%20for%20can%20quickly%20be%20found%20by%20searching%20for%20the%20term%20%E2%80%9CCSI%E2%80%9D%20(see%20where%20the%20title%20of%20this%20post%20comes%20from%3F).%26nbsp%3B%20CSI%20stands%20for%20Component%20Servicing%20Infrastructure%20and%20is%20responsible%20for%20putting%20the%20patch%20files%20on%20the%20system.%26nbsp%3B%20If%20you%20are%20getting%20errors%20due%20to%20permissions%20or%20possibly%20from%20antivirus%20products%2C%20this%20is%20the%20category%20you%20want%20to%20look%20for.%20Don%E2%80%99t%20forget%20to%20search%20in%20the%20%E2%80%9CUp%E2%80%9D%20direction%E2%80%A6gets%20me%20every%20time.%20Note%20the%20line%20below%20found%20while%20searching%20the%20CSI%20messages.%26nbsp%3B%20The%20reference%20to%20Volume%20serial%20number%20as%20well%20as%20the%20name%20of%20the%20target%20file%2C%20which%20references%20a%20group%20policy%20administrative%20template%2C%20were%20instrumental%20fingerprints%20in%20this%20case.%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22image004.png%22%20style%3D%22width%3A%20815px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F273949i29C3F0C8D00D45D0%2Fimage-dimensions%2F815x432%3Fv%3Dv2%22%20width%3D%22815%22%20height%3D%22432%22%20role%3D%22button%22%20title%3D%22image004.png%22%20alt%3D%22image004.png%22%20%2F%3E%3C%2FSPAN%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3ENow%2C%20let%E2%80%99s%20go%20take%20a%20look%20at%20that%20location%20to%20see%20if%20anything%20looks%20wonky.%26nbsp%3B%20The%20first%20thing%20we%20notice%20is%20that%20there%20is%20a%20%3CSTRONG%3E%3CI%3EPolicyDefinitions_old%3C%2FI%3E%3C%2FSTRONG%3E%20folder%2C%20which%20appears%20to%20be%20a%20backup%20of%20the%20unchanged%20folder%E2%80%A6at%20least%20our%20suspect%20is%20cautious.%26nbsp%3B%20Notice%20the%20%E2%80%9CShortcut%E2%80%9D%20icon%20on%20the%20new%20PolicyDefinitions%20folder.%26nbsp%3B%20Things%20are%20starting%20to%20heat%20up%E2%80%A6getting%20warmer.%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22SymLink.png%22%20style%3D%22width%3A%20805px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F273950iDB86384E57B14090%2Fimage-dimensions%2F805x435%3Fv%3Dv2%22%20width%3D%22805%22%20height%3D%22435%22%20role%3D%22button%22%20title%3D%22SymLink.png%22%20alt%3D%22SymLink.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3ETo%20get%20more%20information%20on%20the%20folder%2C%20let%E2%80%99s%20go%20to%20PowerShell%20and%20take%20a%20look%20at%20the%20object%20with%20%3CSTRONG%3E%3CI%3EGet-Item%20C%3A%5CWindows%5CPolicyDefinitions%20%7C%20fl%20*%3C%2FI%3E%3C%2FSTRONG%3E.%26nbsp%3B%20While%20the%3CSTRONG%3E%3CI%3E%20FullName%3C%2FI%3E%3C%2FSTRONG%3E%20field%20looks%20correct%2C%20the%20%3CSTRONG%3E%3CI%3ETarget%20%3C%2FI%3E%3C%2FSTRONG%3Efield%20references%20another%20volume%20entirely.%26nbsp%3B%20This%20is%20what%20is%20causing%20the%20patch%20installation%20to%20fail.%26nbsp%3B%20After%20talking%20to%20the%20server%20owner%2C%20it%20came%20to%20light%20that%20a%20GPO%20management%20tool%20was%20installed%20on%20the%20system%20and%20the%20link%20was%20created%20to%20assist%20the%20product%20manage%20the%20admin%20templates.%26nbsp%3B%20The%20actual%20software%20had%20been%20installed%20on%20a%20different%20volume%2C%20hence%20the%20Junction%20point%20going%20to%20a%20different%20drive.%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Symlink_DirJunctionPoSh.png%22%20style%3D%22width%3A%20807px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F273951i5521F5B3247ED010%2Fimage-dimensions%2F807x492%3Fv%3Dv2%22%20width%3D%22807%22%20height%3D%22492%22%20role%3D%22button%22%20title%3D%22Symlink_DirJunctionPoSh.png%22%20alt%3D%22Symlink_DirJunctionPoSh.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId--503441508%22%20id%3D%22toc-hId--504128684%22%3E%3CFONT%20color%3D%22%230000FF%22%3EThe%20Fix%3A%3C%2FFONT%3E%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3EThe%20fix%20for%20this%20situation%20was%20to%20remove%20the%20Junction%20point%20and%20re-create%20the%20PolicyDefinitions%20folder%20so%20that%20the%20patch%20could%20install%20the%20new%20templates.%26nbsp%3B%20Since%20the%20Junction%20was%20a%20requirement%2C%20we%20had%20to%20copy%20the%20new%20admin%20templates%20to%20the%20target%20folder%20and%20then%20restore%20that%20junction%20point%20temporarily%20but%20are%20actively%20working%20to%20come%20up%20with%20a%20solution%20that%20does%20not%20trigger%20the%20different%20volume%20problem.%3C%2FP%3E%0A%3CP%20class%3D%22MsoNormal%22%3EAnother%20tough%20issue%20put%20to%20bed%20but%20I%E2%80%99m%20sure%20many%20more%20will%20follow.%20%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3C%2FP%3E%0A%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2278566%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%3EA%20mild%20mannered%20IT%20administrator%20is%20doing%20routine%20patching%20when%20things%20go%20south.%26nbsp%3B%20At%20first%20it%20seems%20like%20maybe%20just%20a%20case%20of%20the%20wrong%20patch%20for%20the%20wrong%20system%20but%20there%20is%20more%20to%20this%20story.%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2278566%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EJoelVickery%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎Apr 20 2021 05:56 AM
Updated by: