Hi folks, Ned here again. Occasionally, someone pings me to explain why USMT 4.0 scanstate is running slowly. My first demand is to see the scanstate.log file, having provided the argument /v:5 . That log shows the command-line, the XML files selected, and what "slow" really means.
In this particular example the cause is like Soylent Green: it's peeeeoplllllle!!!
The Scenario and Symptoms
- Scanstate is stuck for a long time at "starting the migration process" on a production computer. Often for many minutes, even though a test computer goes through that same phase in a few seconds.
- The Examination and Gathering phases are as quick as a test computer.
- There are no errors in the console and the migration eventually completes.
- The scanstate.log shows many entries like:
Waiting 6000 msec to retry hive load (tries remaining: 17 )...
IndirectKeyMapper: RegLoadKey(HKEY_USERS,S-1-5-21-2007108519-768573118-610138143-1118, C:\Documents and Settings\bshirley\NTUSER.DAT) failed (3)
Dumping hive list at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\hivelist...
[\REGISTRY\MACHINE\HARDWARE] =>
[\REGISTRY\MACHINE\SECURITY] => \Device\HarddiskVolume1\WINDOWS\system32\config\SECURITY
[\REGISTRY\MACHINE\SOFTWARE] => \Device\HarddiskVolume1\WINDOWS\system32\config\software
[\REGISTRY\MACHINE\SYSTEM] => \Device\HarddiskVolume1\WINDOWS\system32\config\system
[\REGISTRY\USER\.DEFAULT] => \Device\HarddiskVolume1\WINDOWS\system32\config\default
[\REGISTRY\MACHINE\SAM] => \Device\HarddiskVolume1\WINDOWS\system32\config\SAM
[\REGISTRY\USER\S-1-5-20] => \Device\HarddiskVolume1\Documents and Settings\NetworkService\NTUSER.DAT
[\REGISTRY\USER\S-1-5-20_Classes] => \Device\HarddiskVolume1\Documents and Settings\NetworkService\Local Settings\Application Data\Microsoft\Windo
[\REGISTRY\USER\S-1-5-19] => \Device\HarddiskVolume1\Documents and Settings\LocalService\NTUSER.DAT
[\REGISTRY\USER\S-1-5-19_Classes] => \Device\HarddiskVolume1\Documents and Settings\LocalService\Local Settings\Application Data\Microsoft\Windows
[\REGISTRY\USER\S-1-5-21-2007108519-768573118-610138143-500] => \Device\HarddiskVolume1\Documents and Settings\Administrator.CONTOSO\NTUSER.DAT
[\REGISTRY\USER\S-1-5-21-2007108519-768573118-610138143-500_Classes] => \Device\HarddiskVolume1\Documents and Settings\Administrator.CONTOSO\Local
End of hive list
Waiting 6000 msec to retry hive load (tries remaining: 16 )...
IndirectKeyMapper: RegLoadKey(HKEY_USERS,S-1-5-21-2007108519-768573118-610138143-1118, C:\Documents and Settings\bshirley\NTUSER.DAT) failed (3)
- Looking closer, you see 20 of these entries repeated for one or more user profiles.
- After each of those 20 tries, USMT gives up with a message like " Incomplete user profile detected and ignored:C:\Users\ned. Error: 3 "
- Cancelling processing with CTRL+C takes just as long as waiting for the pause to end naturally. You have to kill the scanstate.exe process manually if you want it to stop sooner.
What is the data telling us?
I’ll start Windows Explorer and regedit.exe , and then look at the user profile folder and the registry ProfileList key:
If XP --> C:\Documents and Settings
If Vista/Win7 –> C:\Users
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList
See the issue? My file system looks like this:
But my registry looks like this:
Notice how there are six user SIDs in the ProfileList key, but only three user profiles on the file system. It is supposed to look like this:
Where are Brett Shirley and the others? I'll tell you where: in the bit bucket, because somebody thinks that deleting a folder on the file system is the same as deleting profiles. It's not. That error we kept seeing the log is:
C:\>net helpmsg 3
The system cannot find the path specified.
Why does USMT keep looking?
USMT 4.0 tries to enumerate each profile 20 times, waiting 6 seconds a try, before giving up and moving to the next profile. This design handled a particular third party product that temporarily locked and renamed ntuser.dat files - a user's "HKEY_CURRENT_USER" registry - then changed it back after it was done doing… whatever. You can probably guess what kind of software would do such a thing. You can also do the math now on how long only a few orphaned profiles will delay your migration.
USMT hoped to "wait out" the software and migrate once the profile reappeared. The bad side effect of this error handling is that profiles with no ntuser.dat also cause a wait and retry. The only way to have orphaned profiles is by a human deleting the user profile folders manually instead of using Windows profile tools. There's no bug here with profiles or USMT, it’s all expected behavior.
Don’t you love the law of unintended consequences?
Let's fix it
Luckily, there are a number of easy ways to get your migration back in order. Pick one or more.
- Delete any orphaned profiles using supported tools before running USMT. After all, they point to non-existent data. This can be done a number of ways:
o User Profiles applet within sysdm.cpl (works on all OS; for XP it's the only in-box option)
o Call WMI class Win32_UserProfile to delete profiles on Windows Vista or Windows 7. There are plenty of VB and batch examples on the internet. Here's mine with PowerShell on Win7, where I am zapping all profiles called bshirley:
(Get-WmiObject Win32_UserProfile | Where {$_.LocalPath -like 'c:\users\user'}).Delete()
o Script REG.EXE and RD.EXE (this will only work on Vista or later; the reg.exe included with XP lacks sophisticated search options). For example:
@echo off
REM Commandline needs a deleted user folder to reconcile.
REM Example: RMProf.CMD "c:\users\bshirley"
REG QUERY "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList" /s /d /f %1 | findstr /i "S-1-5-21" > delme.txt
FOR /F "tokens=1* delims=\" %%i in (delme.txt) do REG DELETE "HKLM\%%j" /f
RD %1 /q /s
Del delme.txt /q
o Various third party tools
o Techniques people discuss in the Comments section below. Before you say Delprof.exe , give it a try with a deleted user folder. :)
Then you can run scanstate.exe normally.
- Set the following environment variable in the CMD prompt running scanstate.exe :
SET MIG_IGNORE_PROFILE_MISSING=1
Hardlink migrations haul #%&
This environment variable forces USMT to skip profiles that have no NTUSER.DAT file. This is generally safe to use: Windows Easy Transfer (the consumer-oriented graphical version of USMT) always has this set on internally, so it is well tested and proven. The only downside is that if you ran into that third party software you'd see issues. You would also be skipping profiles that don't contain an ntuser.dat (but do contain a real folder); that's low risk as they are not truly useable profiles anymore either - no one could have logged in with them.
- If using SCCM/MDT to call USMT, the awesome Deployment Guys wrote a script that can detect these orphaned profiles and let you act on them in some fashion. It won't delete these profiles but you now have scriptable options. Setting MIG_IGNORE_PROFILE_MISSING=1 works for MDT/SCCM as well, but automating lite touch/zero touch profile migration to automatically ignore damaged profiles is probably not a great idea; better to detect issues and let a human act up on it.
Update 8-3-2011: Michael Murgolo from the Deployment Guys wrote an awesome sample script that allows you to set environment variables through SCCM/MDT - and the example is for MIG_IGNORE_PROFILE_MISSING! http://blogs.technet.com/b/deploymentguys/archive/2011/08/03/setting-environment-variables-in-a...
Rumor has it that Mike Stephens is working on a series of blog posts about user profile architecture. You can nag him about it here .
Until next time.
Ned "Make Room! Make Room!" Pyle