By
Published
Apr 04 2019 03:17 PM
1,199
Views
First published on TechNet on Dec 16, 2009
Ned here again. Windows Server 2008 introduced a new way to roll out your computers called Windows Deployment Services (WDS). WDS replaces the old Remote Installation Services introduced a decade ago with Windows 2000. By leveraging WDS, you can create image-based deployments, script settings, multicast, and all that other stuff that gets the OS installation folks excited.
If you dig into the WDS documentation you will find that DFSR and DFSN are recommended for scaling out your deployment servers. You can centralize administration and improve performance by creating multiple distribution points that replicate OS images. All while using components you already paid for when you bought your servers.
A few months ago, someone asked if DFSR’s Remote Differential Compression (RDC) feature had any benefits when being used with WDS, where OS images are periodically updated . Would a robocopy script perform better? Are there any interoperability issues? Is the only advantage that files are automatically copied?
I’ve been asked about this a few more times since then, so today I’ll talk about the results. Spoiler alert: WDS and DFSR work really well together. :)
With the invaluable assistance of Scott “the silver fox” McArthur from our sibling Web site AskCore I setup the following repro environment:
WDS uses a single instance store (SIS) as part of its storage mechanism. In basic terms, this means that data common among images is unified in one RWM file. And then the differences are stored in the OS image WIM file, which operates like a stub when you start installing the OS. This saves disk space, which is critical when you’re talking about dozens of images. This also means that both the Install*.WIM and Res.RWM files are altered when modifying your image.
Note: in my case I did not have to increase the staging quota as my files still easily fit in the 4GB default staging space. You may need to increase it using our old, reliable DFSR performance tuning article:
http://blogs.technet.com/b/askds/archive/2010/03/31/tuning-replication-performance-in-dfsr-espec...
WDS always replaces the existing Install*.WIM file with another copy. Additionally, the binary contents of the replaced file are effectively completely different than the original – it’s just how WIM creation works. So RDC does not provide any gains when replicating a WIM file. The file is copied in its entirety, but more on this below. However, the WIM files in WDS are comparatively small, so DFSR moves them very fast.
WDS does not replace RWM files; it modifies them in place. This means that DFSR uses RDC and we see substantial savings over the wire in bandwidth. In our test, we added a small amount of data to the mounted image, which modifies the Res.RWM file. DFSR dies not replicate the whole 3.6GB of data; instead, DFSR replicates the delta of the change plus some overhead. The total delta of replicated data with the RWM file was around 1MB of data. That is an RDC savings of roughly 99.98%.
The cost here when using RDC, is that a file this large spends most of its time in local disk I/O and CPU time on both WDS servers.
So only a few seconds and 1MB of data was network time – the rest is “thinking time”. If you have a low-latency, high-bandwidth, gigabit network between all servers, then using DFSR with RDC to save bandwidth is not advantageous over just copying the files with say robocopy or xcopy – it might appear slightly slower. But if the network WAN connections are not fast or high bandwidth, then DFSR with RDC saves a ton of bandwidth and time. Also, you can turn off RDC on just these super-fast network scenarios and at least get DFSR’s “keep my files in sync so I don’t have to think about it” advantage.
That’s the short answer. Not everyone likes to read my DFSR debug diatribes , but if you do, carry on to the next section.
20091007 18:06:13.330 2680 USNC 2450 UsnConsumer::UpdateIdRecord LDB Updating ID Record: ß file is modified, not replaced
+ fid 0x30000000000B0
+ usn 0x6b480
+ uidVisible 1
+ filtered 0
+ journalWrapped 0
+ slowRecoverCheck 0
+ pendingTombstone 0
+ internalUpdate 0
+ dirtyShutdownMismatch 0
+ meetInstallUpdate 0
+ meetReanimated 0
+ recUpdateTime 20091007 21:29:29.115 GMT
+ present 1
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:13.298 GMT (0x1ca479a624956e1)
+ createTime 20091007 18:46:43.088 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 00000000-00000000-00000000-00000000
+ similarity 00000000-00000000-00000000-00000000
+ name Res.RWM ß the RWM file
20091007 18:06:13.330 2680 USNC 2453 UsnConsumer::UpdateIdRecord ID record updated from USN_RECORD: ß USN update showing modification
+ USN_RECORD:
+ RecordLength: 80
+ MajorVersion: 2
+ MinorVersion: 0
+ FileRefNumber: 0x30000000000B0
+ ParentFileRefNumber: 0x10000000000C7
+ USN: 0x6b480
+ TimeStamp: 20091007 18:06:13.298 Eastern Standard Time
+ Reason: Close Data Extend Data Overwrite ß modifying here, not replacing
+ SourceInfo: 0x0
+ SecurityId: 0x0
+ FileAttributes: 0x2020
+ FileNameLength: 14
+ FileNameOffset: 60
+ FileName: Res.RWM ß the RWM file
20091007 18:06:13.689 2808 STAG 4257 Staging::GetStageReaderOrWriter ß file gets staged
+ fid 0x30000000000B0
+ usn 0x6b480
+ uidVisible 1
+ filtered 0
+ journalWrapped 0
+ slowRecoverCheck 0
+ pendingTombstone 0
+ internalUpdate 0
+ dirtyShutdownMismatch 0
+ meetInstallUpdate 0
+ meetReanimated 0
+ recUpdateTime 20091007 22:06:13.330 GMT
+ present 1
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:13.298 GMT (0x1ca479a624956e1)
+ createTime 20091007 18:46:43.088 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 00000000-00000000-00000000-00000000
+ similarity 00000000-00000000-00000000-00000000
+ name Res.RWM ß the RWM file
+ StageReader:0000000000000000 StageWriter:000000000032A1B0 Policy:1
20091007 18:06:14.064 2808 RDCX 467 StreamToIndex RDC generate begin: (0..4), uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 fileName:Res.RWM csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} ß RDC signature calculation starts
20091007 18:09:49.903 2808 RDCX 509 StreamToIndex RDC generate end: (0..4), uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 fileName:Res.RWM csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} ß RDC signatures calculation done (took ~ 3 minutes 35 seconds)
20091007 18:09:50.341 2808 SRTR 2592 InitializeFileTransferAsyncState::~InitializeFileTransferAsyncState this:000000000032FD00
20091007 18:14:37.972 2228 OUTC 3273 OutConnectionContentSetContext::SERVER_RdcGetSignatures Sent requested RDC signatures. context:00000000003BE770 rdcState:00000000003194B0 reader:0000000000000000 level:1 offset:32826220 length:422 sizeRead:422 uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} ß signature data gets used to creates packages of data and is sent – this takes ~ 4 minutes 15 seconds comparing to signatures downstream and sending the data
20091007 18:14:44.113 2228 RDCX 2927 Rdc::SyncServerState::~SyncServerState RDC Need Reader Statistics: uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ TOTAL
+ Compression Ratio 31
+ RDC Need Size 1381790
+ Bytes sent to downstream 958704 ß Only about 1MB (!!!) of data from the 3756455865 byte file actually had to be sent to B though. Very cool RDC savings!
+ Uncompressed XPRESS blocks 0
+ Compressed XPRESS blocks 4
+ Copied XPRESS Blocks 1
+ Bytes read using async I/Os 914469
Server B:
20091007 18:14:43.826 324 XRNA 276 XpressRdcNeedAssembler::ProcessRdcNeeds All needs processed. this:0000000017384710
20091007 18:14:43.826 324 XRNA 291 XpressRdcNeedAssembler::FinalizeRpcDownload Shutting down RPC pipe thread this:0000000017384710
20091007 18:14:43.826 1436 XRNA 919 XpressRdcNeedAssembler::GetSourceRdcNeedDataThread Exiting RDC source need download thread. this:0000000017384710 ß finished getting the files bits
20091007 18:14:43.873 324 STAG 798 StageWriter::CompleteDownloadStage Completed download or stage file 241-{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138-{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241-Downloaded.frx
20091007 18:18:08.843 324 ASYN 508 AsyncUnbufferedFileWriter::Close Async WRITE Statistics: ß all bits put together, ready to drop into real replicated folder
20091007 18:18:08.843 324 MEET 3013 Meet::InstallRename Moving contents from Installing to final destination. Attributes:0x2020 updateName:Res.RWM uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csName:RemoteInstall ß done. Took ~12 minutes of “real time”, but you saved 3.6 GB on the wire(!!!)
=======================================
INSTALL.WIM behavior:
Server A:
20091007 18:06:12.830 2680 USNC 2881 UsnConsumer::TombstoneOrDelete LDB Updating ID Record: ß install.wim is deleted and database record updated
+ fid 0x500000000006A
+ usn 0x6b388
+ uidVisible 1
+ filtered 0
+ journalWrapped 0
+ slowRecoverCheck 0
+ pendingTombstone 0
+ internalUpdate 0
+ dirtyShutdownMismatch 0
+ meetInstallUpdate 0
+ meetReanimated 0
+ recUpdateTime 20091007 21:29:31.255 GMT
+ present 0
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v240
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:12.642 GMT (0x1ca479a61e532f7)
+ createTime 20091007 18:47:25.714 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 5DB57C5A-C50E04D8-DEAAFED0-04B337A2
+ similarity 3E311339-251E183B-20271C15-2C183010
+ name install.wim ß the WIM
20091007 18:06:13.455 2680 USNC 3490 UsnConsumer::UidTunnelReanimate LDB Updating ID Record: ß same named/pathed file was written in (aka the delete above was really an overwrite)
+ fid 0x600000000006A
+ usn 0x6b5d8
+ uidVisible 1
+ filtered 0
+ journalWrapped 0
+ slowRecoverCheck 0
+ pendingTombstone 0
+ internalUpdate 0
+ dirtyShutdownMismatch 0
+ meetInstallUpdate 0
+ meetReanimated 0
+ recUpdateTime 20091007 22:06:12.830 GMT
+ present 1
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v242
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:13.408 GMT (0x1ca479a625a0788)
+ createTime 20091007 18:47:25.714 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 00000000-00000000-00000000-00000000
+ similarity 00000000-00000000-00000000-00000000
+ name install.wim ß the WIM
20091007 18:06:13.783 2680 USNC 2450 UsnConsumer::UpdateIdRecord LDB Updating ID Record: ß WDS writes some more data to the file
+ fid 0x600000000006A
+ usn 0x6b6e0
+ uidVisible 1
+ filtered 0
+ journalWrapped 0
+ slowRecoverCheck 0
+ pendingTombstone 0
+ internalUpdate 0
+ dirtyShutdownMismatch 0
+ meetInstallUpdate 0
+ meetReanimated 0
+ recUpdateTime 20091007 22:06:13.752 GMT
+ present 1
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v244 ß note how we’re at 244, so it’s been modified a bit more by WDS
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:13.752 GMT (0x1ca479a628e7bdf)
+ createTime 20091007 18:47:25.714 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 00000000-00000000-00000000-00000000
+ similarity 00000000-00000000-00000000-00000000
+ name install.wim
20091007 18:06:13.783 2680 USNC 2453 UsnConsumer::UpdateIdRecord ID record updated from USN_RECORD:
+ USN_RECORD:
+ RecordLength: 88
+ MajorVersion: 2
+ MinorVersion: 0
+ FileRefNumber: 0x600000000006A
+ ParentFileRefNumber: 0x10000000000C7
+ USN: 0x6b6e0
+ TimeStamp: 20091007 18:06:13.752 Eastern Standard Time
+ Reason: Close Data Extend Data Overwrite ß data added
+ SourceInfo: 0x0
+ SecurityId: 0x0
+ FileAttributes: 0x2020
+ FileNameLength: 22
+ FileNameOffset: 60
+ FileName: install.wim
20091007 18:06:16.502 556 RDCX 2927 Rdc::SyncServerState::~SyncServerState RDC Need Reader Statistics: uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v245 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ TOTAL
+ Compression Ratio 0
+ RDC Need Size 2660306
+ Bytes sent to downstream 2682260 ß we have to send these many bytes to server B. There is no RDC saving here, the file was scrambled by WIM format.
+ Uncompressed XPRESS blocks 0
+ Compressed XPRESS blocks 0
+ Copied XPRESS Blocks 2
+ Bytes read using async I/Os 2656520
20091007 18:06:16.517 556 ASYN 1289 AsyncUnbufferedFileReader::Close Async READ Statistics:
+ Total I/O Bytes 2656520 (3 buffers)
+ Number of Async Buffers 2
+ Size of Async Buffers 1048576
+ Number of Async I/Os 3
+ Number of Sync I/Os 0
+ Number of Pending Calls 1
+ Number of Queue Depth Empty 0
+ Number of Buffers Used 2
On Server B:
20091007 18:06:18.139 1852 INCO 6593 InConnection::LogTransferActivity Received RAWGET uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v246 fileName:install.wim connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} stagedSize:2657686 ß RDC could not be used – no matching signatures - so we do a raw copy
20091007 18:06:18.545 1852 INCO 4825 InConnection::UpdateProcessed Received Update. updatesLeft:0 processed:1 failures:0 sessionId:6 open:0 updateType:0 processStatus:0 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} csName:RemoteInstall update: ß final version of file replicated
+ present 1
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v245
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:13.767 GMT (0x1ca479a6290de3f)
+ createTime 20091007 18:47:25.714 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 2BAC2FE6-3735E3A6-F89F8423-37C51923
+ similarity 1D2A1A21-262A310C-0C060426-06291D0B
+ name install.wim
20091007 18:06:18.498 1852 MEET 1804 Meet::InstallStep Done installing file updateName:install.wim uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v246 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csName:RemoteInstall ß B has the new file. Took about 6 seconds on a fast network so this would be much slower on most WAN's. And since the file was never that big anyways, probably acceptable behavior.
So there it is. If you’re using WDS and DFSR and your servers are not connected purely by gigabit end-to-end, modifying your images and letting DFSR take care of the modification can save big time and bandwidth. And that is something you can put towards your bottom line with more productive users who can use more of that WAN for other things.
- Ned “the matchmaker” Pyle
Ned here again. Windows Server 2008 introduced a new way to roll out your computers called Windows Deployment Services (WDS). WDS replaces the old Remote Installation Services introduced a decade ago with Windows 2000. By leveraging WDS, you can create image-based deployments, script settings, multicast, and all that other stuff that gets the OS installation folks excited.
If you dig into the WDS documentation you will find that DFSR and DFSN are recommended for scaling out your deployment servers. You can centralize administration and improve performance by creating multiple distribution points that replicate OS images. All while using components you already paid for when you bought your servers.
A few months ago, someone asked if DFSR’s Remote Differential Compression (RDC) feature had any benefits when being used with WDS, where OS images are periodically updated . Would a robocopy script perform better? Are there any interoperability issues? Is the only advantage that files are automatically copied?
I’ve been asked about this a few more times since then, so today I’ll talk about the results. Spoiler alert: WDS and DFSR work really well together. :)
The Test Environment
With the invaluable assistance of Scott “the silver fox” McArthur from our sibling Web site AskCore I setup the following repro environment:
- Two Windows Server 2008 R2 WDS servers
- Both servers running DFSR with hotfix 975763 to update DFSRS.EXE.
- Server “A” is the WDS administrative server where all management work is done
- Server “B” is a WDS branch server that simply accepts whatever is sent to him and is used to deploy computers remotely
- A variety of Win7/R2 family WIM files in an install image group, plus their single instance store RWM file. More on this in a bit.
-
DFSR replication is configured as described in article
Storing and Replicating Images Using DFS
, ensuring a
DFSR filter
is configured to exclude the
RemoteInstall
subfolders:
- \Mgmt
- \Tmp
- \Stores (If Windows Server 2008 R2)
- Using WDS methodology to:
- Export an image
- Mount it
- Make changes to it ( in my case, about ~1MB of data added to the image)
- Dismount it
- Replace the previous image
WDS uses a single instance store (SIS) as part of its storage mechanism. In basic terms, this means that data common among images is unified in one RWM file. And then the differences are stored in the OS image WIM file, which operates like a stub when you start installing the OS. This saves disk space, which is critical when you’re talking about dozens of images. This also means that both the Install*.WIM and Res.RWM files are altered when modifying your image.
Note: in my case I did not have to increase the staging quota as my files still easily fit in the 4GB default staging space. You may need to increase it using our old, reliable DFSR performance tuning article:
http://blogs.technet.com/b/askds/archive/2010/03/31/tuning-replication-performance-in-dfsr-espec...
The Results
WDS always replaces the existing Install*.WIM file with another copy. Additionally, the binary contents of the replaced file are effectively completely different than the original – it’s just how WIM creation works. So RDC does not provide any gains when replicating a WIM file. The file is copied in its entirety, but more on this below. However, the WIM files in WDS are comparatively small, so DFSR moves them very fast.
WDS does not replace RWM files; it modifies them in place. This means that DFSR uses RDC and we see substantial savings over the wire in bandwidth. In our test, we added a small amount of data to the mounted image, which modifies the Res.RWM file. DFSR dies not replicate the whole 3.6GB of data; instead, DFSR replicates the delta of the change plus some overhead. The total delta of replicated data with the RWM file was around 1MB of data. That is an RDC savings of roughly 99.98%.
The cost here when using RDC, is that a file this large spends most of its time in local disk I/O and CPU time on both WDS servers.
- Upstream on “A” - copying the file to staging, walking all of its data to create signatures – took several minutes
- Downstream on B – copying the file to staging (unless it already exists), walking all of its data to create signatures, taking the bits that were sent over the wire and reconstructing the file with its changes, installing the file
So only a few seconds and 1MB of data was network time – the rest is “thinking time”. If you have a low-latency, high-bandwidth, gigabit network between all servers, then using DFSR with RDC to save bandwidth is not advantageous over just copying the files with say robocopy or xcopy – it might appear slightly slower. But if the network WAN connections are not fast or high bandwidth, then DFSR with RDC saves a ton of bandwidth and time. Also, you can turn off RDC on just these super-fast network scenarios and at least get DFSR’s “keep my files in sync so I don’t have to think about it” advantage.
That’s the short answer. Not everyone likes to read my DFSR debug diatribes , but if you do, carry on to the next section.
Seeing it all under the covers
20091007 18:06:13.330 2680 USNC 2450 UsnConsumer::UpdateIdRecord LDB Updating ID Record: ß file is modified, not replaced
+ fid 0x30000000000B0
+ usn 0x6b480
+ uidVisible 1
+ filtered 0
+ journalWrapped 0
+ slowRecoverCheck 0
+ pendingTombstone 0
+ internalUpdate 0
+ dirtyShutdownMismatch 0
+ meetInstallUpdate 0
+ meetReanimated 0
+ recUpdateTime 20091007 21:29:29.115 GMT
+ present 1
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:13.298 GMT (0x1ca479a624956e1)
+ createTime 20091007 18:46:43.088 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 00000000-00000000-00000000-00000000
+ similarity 00000000-00000000-00000000-00000000
+ name Res.RWM ß the RWM file
20091007 18:06:13.330 2680 USNC 2453 UsnConsumer::UpdateIdRecord ID record updated from USN_RECORD: ß USN update showing modification
+ USN_RECORD:
+ RecordLength: 80
+ MajorVersion: 2
+ MinorVersion: 0
+ FileRefNumber: 0x30000000000B0
+ ParentFileRefNumber: 0x10000000000C7
+ USN: 0x6b480
+ TimeStamp: 20091007 18:06:13.298 Eastern Standard Time
+ Reason: Close Data Extend Data Overwrite ß modifying here, not replacing
+ SourceInfo: 0x0
+ SecurityId: 0x0
+ FileAttributes: 0x2020
+ FileNameLength: 14
+ FileNameOffset: 60
+ FileName: Res.RWM ß the RWM file
20091007 18:06:13.689 2808 STAG 4257 Staging::GetStageReaderOrWriter ß file gets staged
+ fid 0x30000000000B0
+ usn 0x6b480
+ uidVisible 1
+ filtered 0
+ journalWrapped 0
+ slowRecoverCheck 0
+ pendingTombstone 0
+ internalUpdate 0
+ dirtyShutdownMismatch 0
+ meetInstallUpdate 0
+ meetReanimated 0
+ recUpdateTime 20091007 22:06:13.330 GMT
+ present 1
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:13.298 GMT (0x1ca479a624956e1)
+ createTime 20091007 18:46:43.088 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 00000000-00000000-00000000-00000000
+ similarity 00000000-00000000-00000000-00000000
+ name Res.RWM ß the RWM file
+ StageReader:0000000000000000 StageWriter:000000000032A1B0 Policy:1
20091007 18:06:14.064 2808 RDCX 467 StreamToIndex RDC generate begin: (0..4), uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 fileName:Res.RWM csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} ß RDC signature calculation starts
20091007 18:09:49.903 2808 RDCX 509 StreamToIndex RDC generate end: (0..4), uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 fileName:Res.RWM csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} ß RDC signatures calculation done (took ~ 3 minutes 35 seconds)
20091007 18:09:50.341 2808 SRTR 2592 InitializeFileTransferAsyncState::~InitializeFileTransferAsyncState this:000000000032FD00
20091007 18:14:37.972 2228 OUTC 3273 OutConnectionContentSetContext::SERVER_RdcGetSignatures Sent requested RDC signatures. context:00000000003BE770 rdcState:00000000003194B0 reader:0000000000000000 level:1 offset:32826220 length:422 sizeRead:422 uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} ß signature data gets used to creates packages of data and is sent – this takes ~ 4 minutes 15 seconds comparing to signatures downstream and sending the data
20091007 18:14:44.113 2228 RDCX 2927 Rdc::SyncServerState::~SyncServerState RDC Need Reader Statistics: uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ TOTAL
+ Compression Ratio 31
+ RDC Need Size 1381790
+ Bytes sent to downstream 958704 ß Only about 1MB (!!!) of data from the 3756455865 byte file actually had to be sent to B though. Very cool RDC savings!
+ Uncompressed XPRESS blocks 0
+ Compressed XPRESS blocks 4
+ Copied XPRESS Blocks 1
+ Bytes read using async I/Os 914469
Server B:
20091007 18:14:43.826 324 XRNA 276 XpressRdcNeedAssembler::ProcessRdcNeeds All needs processed. this:0000000017384710
20091007 18:14:43.826 324 XRNA 291 XpressRdcNeedAssembler::FinalizeRpcDownload Shutting down RPC pipe thread this:0000000017384710
20091007 18:14:43.826 1436 XRNA 919 XpressRdcNeedAssembler::GetSourceRdcNeedDataThread Exiting RDC source need download thread. this:0000000017384710 ß finished getting the files bits
20091007 18:14:43.873 324 STAG 798 StageWriter::CompleteDownloadStage Completed download or stage file 241-{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138-{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241-Downloaded.frx
20091007 18:18:08.843 324 ASYN 508 AsyncUnbufferedFileWriter::Close Async WRITE Statistics: ß all bits put together, ready to drop into real replicated folder
20091007 18:18:08.843 324 MEET 3013 Meet::InstallRename Moving contents from Installing to final destination. Attributes:0x2020 updateName:Res.RWM uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csName:RemoteInstall ß done. Took ~12 minutes of “real time”, but you saved 3.6 GB on the wire(!!!)
=======================================
INSTALL.WIM behavior:
Server A:
20091007 18:06:12.830 2680 USNC 2881 UsnConsumer::TombstoneOrDelete LDB Updating ID Record: ß install.wim is deleted and database record updated
+ fid 0x500000000006A
+ usn 0x6b388
+ uidVisible 1
+ filtered 0
+ journalWrapped 0
+ slowRecoverCheck 0
+ pendingTombstone 0
+ internalUpdate 0
+ dirtyShutdownMismatch 0
+ meetInstallUpdate 0
+ meetReanimated 0
+ recUpdateTime 20091007 21:29:31.255 GMT
+ present 0
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v240
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:12.642 GMT (0x1ca479a61e532f7)
+ createTime 20091007 18:47:25.714 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 5DB57C5A-C50E04D8-DEAAFED0-04B337A2
+ similarity 3E311339-251E183B-20271C15-2C183010
+ name install.wim ß the WIM
20091007 18:06:13.455 2680 USNC 3490 UsnConsumer::UidTunnelReanimate LDB Updating ID Record: ß same named/pathed file was written in (aka the delete above was really an overwrite)
+ fid 0x600000000006A
+ usn 0x6b5d8
+ uidVisible 1
+ filtered 0
+ journalWrapped 0
+ slowRecoverCheck 0
+ pendingTombstone 0
+ internalUpdate 0
+ dirtyShutdownMismatch 0
+ meetInstallUpdate 0
+ meetReanimated 0
+ recUpdateTime 20091007 22:06:12.830 GMT
+ present 1
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v242
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:13.408 GMT (0x1ca479a625a0788)
+ createTime 20091007 18:47:25.714 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 00000000-00000000-00000000-00000000
+ similarity 00000000-00000000-00000000-00000000
+ name install.wim ß the WIM
20091007 18:06:13.783 2680 USNC 2450 UsnConsumer::UpdateIdRecord LDB Updating ID Record: ß WDS writes some more data to the file
+ fid 0x600000000006A
+ usn 0x6b6e0
+ uidVisible 1
+ filtered 0
+ journalWrapped 0
+ slowRecoverCheck 0
+ pendingTombstone 0
+ internalUpdate 0
+ dirtyShutdownMismatch 0
+ meetInstallUpdate 0
+ meetReanimated 0
+ recUpdateTime 20091007 22:06:13.752 GMT
+ present 1
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v244 ß note how we’re at 244, so it’s been modified a bit more by WDS
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:13.752 GMT (0x1ca479a628e7bdf)
+ createTime 20091007 18:47:25.714 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 00000000-00000000-00000000-00000000
+ similarity 00000000-00000000-00000000-00000000
+ name install.wim
20091007 18:06:13.783 2680 USNC 2453 UsnConsumer::UpdateIdRecord ID record updated from USN_RECORD:
+ USN_RECORD:
+ RecordLength: 88
+ MajorVersion: 2
+ MinorVersion: 0
+ FileRefNumber: 0x600000000006A
+ ParentFileRefNumber: 0x10000000000C7
+ USN: 0x6b6e0
+ TimeStamp: 20091007 18:06:13.752 Eastern Standard Time
+ Reason: Close Data Extend Data Overwrite ß data added
+ SourceInfo: 0x0
+ SecurityId: 0x0
+ FileAttributes: 0x2020
+ FileNameLength: 22
+ FileNameOffset: 60
+ FileName: install.wim
20091007 18:06:16.502 556 RDCX 2927 Rdc::SyncServerState::~SyncServerState RDC Need Reader Statistics: uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v245 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ TOTAL
+ Compression Ratio 0
+ RDC Need Size 2660306
+ Bytes sent to downstream 2682260 ß we have to send these many bytes to server B. There is no RDC saving here, the file was scrambled by WIM format.
+ Uncompressed XPRESS blocks 0
+ Compressed XPRESS blocks 0
+ Copied XPRESS Blocks 2
+ Bytes read using async I/Os 2656520
20091007 18:06:16.517 556 ASYN 1289 AsyncUnbufferedFileReader::Close Async READ Statistics:
+ Total I/O Bytes 2656520 (3 buffers)
+ Number of Async Buffers 2
+ Size of Async Buffers 1048576
+ Number of Async I/Os 3
+ Number of Sync I/Os 0
+ Number of Pending Calls 1
+ Number of Queue Depth Empty 0
+ Number of Buffers Used 2
On Server B:
20091007 18:06:18.139 1852 INCO 6593 InConnection::LogTransferActivity Received RAWGET uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v246 fileName:install.wim connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} stagedSize:2657686 ß RDC could not be used – no matching signatures - so we do a raw copy
20091007 18:06:18.545 1852 INCO 4825 InConnection::UpdateProcessed Received Update. updatesLeft:0 processed:1 failures:0 sessionId:6 open:0 updateType:0 processStatus:0 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} csName:RemoteInstall update: ß final version of file replicated
+ present 1
+ nameConflict 0
+ attributes 0x2020
+ ghostedHeader 0
+ data 0
+ gvsn {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v245
+ uid {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+ parent {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+ fence Default (3)
+ clockDecrementedInDirtyShutdown 0
+ clock 20091007 22:06:13.767 GMT (0x1ca479a6290de3f)
+ createTime 20091007 18:47:25.714 GMT
+ csId {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+ hash 2BAC2FE6-3735E3A6-F89F8423-37C51923
+ similarity 1D2A1A21-262A310C-0C060426-06291D0B
+ name install.wim
20091007 18:06:18.498 1852 MEET 1804 Meet::InstallStep Done installing file updateName:install.wim uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v246 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csName:RemoteInstall ß B has the new file. Took about 6 seconds on a fast network so this would be much slower on most WAN's. And since the file was never that big anyways, probably acceptable behavior.
So there it is. If you’re using WDS and DFSR and your servers are not connected purely by gigabit end-to-end, modifying your images and letting DFSR take care of the modification can save big time and bandwidth. And that is something you can put towards your bottom line with more productive users who can use more of that WAN for other things.
- Ned “the matchmaker” Pyle