Home
%3CLINGO-SUB%20id%3D%22lingo-sub-623910%22%20slang%3D%22en-US%22%3ERe%3A%20Enabling%20Data%20Deduplication%20on%20Windows%20Server%202019%20Volumes%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-623910%22%20slang%3D%22en-US%22%3E%3CP%3EJust%20curious%2C%20what's%20the%20data%20compression%20ratio%20in%20this%20case%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-623981%22%20slang%3D%22en-US%22%3ERe%3A%20Enabling%20Data%20Deduplication%20on%20Windows%20Server%202019%20Volumes%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-623981%22%20slang%3D%22en-US%22%3E%3CP%3EHey%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F243317%22%20target%3D%22_blank%22%3E%40Kexy%20Biscuit%3C%2FA%3E%26nbsp%3B%20-%20the%20compression%20ratio%20will%20vary%20based%20on%20the%20type%20of%20files%20in%20use.%20I'll%20update%20the%20post%20once%20later%20this%20week%20once%20the%20data%20transfer%20and%20deduplication%20jobs%20have%20run%20with%20the%20production%20workload.%26nbsp%3B%20My%20%22guestimate%22%20is%20not%20that%20high%20for%20unique%20files%20as%20they%20are%20video%20files%20-%20but%20actual%20duplicate%20files%20due%20to%20user%20error%20or%20filing%20mishaps%20will%20be%20biggest%20winners.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-623521%22%20slang%3D%22en-US%22%3EEnabling%20Data%20Deduplication%20on%20Windows%20Server%202019%20Volumes%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-623521%22%20slang%3D%22en-US%22%3E%3CP%3EI%20was%20recently%20asked%20to%20help%20out%20our%20friends%20at%20Channel%209%20with%20a%20storage%20problem.%20They%20capture%20large%20video%20files%20as%20part%20of%20their%20recordings%20and%20have%20to%20have%20them%20local%20for%20fast%20access%20in%20order%20to%20edit%20them.%20They%20currently%20use%20a%20proprietary%20Linux%20box%20for%20NFS%20storage%20which%20is%20just%20about%20full.%20I%20have%20spec'ed%20out%20a%20Storage%20Spaces%20Direct%20setup%20on%20Windows%202019%20from%20the%20%3CA%20href%3D%22https%3A%2F%2Fwww.microsoft.com%2Fcloud-platform%2Fazure-stack-hci-catalog%3FWT.mc_id%3Ditopstalk-blog-rclaus%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAzure%20Stack%20HCI%20catalog%3C%2FA%3E%20-%20but%20it%20won't%20be%20in%20for%20a%20while.%20This%20is%20the%20first%20bare%20metal%20system%20I've%20had%20to%20rack%20and%20stack%20in%20a%20while%20and%20it's%20part%20of%20a%20longer%20project%20-%20so%20I%20thought%20it%20would%20be%20good%20to%20write%20up%20a%20short%20set%20of%20topical%20blog%20posts%20to%20go%20along%20with%20the%20process.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CFONT%20size%3D%224%22%3E%3CSTRONG%3EFinding%20the%20right%20physical%20server%20from%20on-premises%20inventory%3C%2FSTRONG%3E%3C%2FFONT%3E%3C%2FP%3E%0A%3CP%3EFirst%20off%20-%20I%20am%20spoiled%20by%20running%20a%20simple%20Azure%20command%20to%20provision%20whatever%20size%20box%20I%20need%20in%20a%20matter%20of%20minutes%20-%20this%20is%20going%20back%20in%20time%20for%20me.%20Luckily%2C%20I%20have%20an%20old%20DataOn%20Cluster-in-a-box%20system%20from%20a%20lab%20that%20has%20some%20decent%20(70%20TB)%20worth%20of%20storage%20to%20temporarily%20hold%20them%20over%20until%20the%20new%20system%20arrives.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-left%22%20style%3D%22width%3A%20300px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F115541i655BEE79DF29F037%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20alt%3D%22DataOnBox.JPG%22%20title%3D%22DataOnBox.JPG%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ELike%20a%20fresh%20coat%20of%20paint%20-%20flattening%20and%20rebuilding%20the%20servers%20from%20a%20USB%20media%20was%20kind%20of%20refreshing.%20In%20no%20time%20at%20all%20I%20have%20two%20nodes%20of%20a%20Windows%20Server%202019%20up%20and%20running.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CFONT%20size%3D%224%22%3E%3CSTRONG%3EInstalling%20Data%20Deduplication%3C%2FSTRONG%3E%3C%2FFONT%3E%3CBR%20%2F%3EThe%20files%20they%20are%20storing%20includes%20large%20video%20files%20copied%20up%20to%20the%20system%20by%20various%20camera%20operators%20in%20the%20field.%20There%20isn't%20a%20sophisticated%20logging%20system%20on%20what%20file%20came%20where%20or%20if%20files%20have%20already%20been%20dumped%20up%20on%20the%20share.%20As%20a%20result%20-%20we've%20got%20duplicates.%20I%20figured%20one%20easy%20way%20of%20resolving%20this%20space%20issue%20is%20to%20enable%20deduplication.%20Data%20Deduplication%20in%20Windows%20Server%202019%20is%20more%20efficient%20than%20previous%20releases%20-%20it%20can%20run%20in%20the%20background%20on%20multiple%20volumes%20with%20minimal%20impact%20on%20other%20workloads.%20If%20you%20want%20some%20more%20details%20on%20Deduplication%20-%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fwindows-server%2Fstorage%2Fdata-deduplication%2Finstall-enable%3FWT.mc_id%3Ditopstalk-blog-rclaus%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ewe've%20got%20a%20good%20article%20over%20on%20Microsoft%20Docs%20you%20should%20check%20out%3C%2FA%3E.%26nbsp%3B%3CBR%20%2F%3EThe%20temporary%20server%20in%20question%20has%20the%20fresh%20install%20of%20Windows%20Server%202019%20with%20all%20available%20drives%20configured%20in%20a%20storage%20pool%20with%20a%20single%20volume.%20I%20really%20just%20needed%20to%20get%20this%20thing%20up%2C%20enable%20deduplication%20and%20copy%20over%20the%20data%20from%20the%20Linux%20box%20until%20the%20new%20AzureStack%20HCI%20configuration%20comes%20in.%20%3CBR%20%2F%3EThe%20fastest%20way%20to%20install%20the%20DataDeduplication%20feature%20is%20to%20crack%20open%20a%20PowerShell%20window%20and%20type%20in%20%3CSTRONG%3EAdd-WindowsFeature%20-name%20FS-Data-Deduplication%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20915px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F115546iB28D722E47460FD0%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22InstallDataDedup.png%22%20title%3D%22InstallDataDedup.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CBR%20%2F%3E%3CFONT%20size%3D%224%22%3E%3CSTRONG%3EEnabling%20and%20Configuring%20Data%20Deduplication%20on%20Data%20Drive%3C%2FSTRONG%3E%3C%2FFONT%3E%3CBR%20%2F%3ETo%20enable%20the%20default%20settings%20for%20DataDeduplication%20for%20my%20E%3A%20drive-%20it's%20was%20simple%20as%20typing%20in%20%3CSTRONG%3EEnable-DedupVolume%20E%3A%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F115548i9609A2D8C981AE0C%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22enableDedup.png%22%20title%3D%22enableDedup.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EBecause%20this%20is%20a%20temporary%20server%20-%20I%20want%20to%20configure%20the%20minimum%20file%20age%20to%20be%20%220%22%20in%20order%20for%20the%20server%20to%20start%20deduplicating%20files%20on%20the%20next%20job%20schedule%20or%20manual%20kickoff.%20To%20change%20this%20parameter%20-%20type%20in%20%3CSTRONG%3ESet-DedupVolume%20-Volume%20%22E%3A%22%20-MinimumFileAgeDays%200%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CBR%20%2F%3E%3CFONT%20size%3D%224%22%3E%3CSTRONG%3ETime%20to%20Finish%20Rack%20and%20Stack%20%2F%20Install%3C%2FSTRONG%3E%3C%2FFONT%3E%3C%2FP%3E%0A%3CP%3EAt%20this%20point%20we're%20all%20ready%20to%20go.%20It's%20a%20matter%20of%20transporting%20and%20installing%20this%20box%20into%20it's%20temporary%20home%20and%20transferring%20the%20files%20over%20to%20it%20from%20the%20Linux%20NAS%20box.%20But%20for%20this%20-%20you'll%20have%20to%20wait%20until%20the%20next%20post%20in%20this%20short%20series.%3C%2FP%3E%0A%3CP%3E%3CBR%20%2F%3EFrom%20start%20to%20finish%20-%20this%20took%20the%20better%20part%20of%20an%20afternoon%20to%20provision%20some%20bare%20metal%2C%20with%2070%20TB%20of%20storage%20as%20a%20temporary%20server.%20That%20is%20a%20tad%20longer%20then%20what%20seems%20like%20a%20very%20long%205%20minutes%20to%20provision%20a%20system%20in%20Azure%20-%20but%20the%20proximity%20and%20fast%20access%20to%20these%20source%20files%20is%20what%20is%20important%20to%20the%20team%20at%20Channel%209.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAll%20this%20being%20said%20-%20as%20I%20was%20going%20through%20this%20-%20I%20was%20thinking%20of%20all%20the%20value-add%20services%20from%20Azure%20I%20could%20bring%20to%20this%20on-prem%20bare%20metal%20box.%20Once%20the%20more%20permanent%20solution%20is%20in%20place%20-%20I'll%20go%20through%20the%20process%20of%20adding%20on%20a%20few%20of%20these%20Azure%20services%20to%20help%20me%20out%3A%20Azure%20File%20Sync%2C%20Azure%20Backup%20and%20a%20bunch%20of%20the%20Management%20add-ons%20to%20monitor%20the%20health%20of%20the%20system%20and%20so%20on.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhat%20about%20you%3F%20Have%20you%20considered%20or%20started%20to%20explore%20any%20of%20the%20Hybrid%20Azure-to-on-prem%20services%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-623521%22%20slang%3D%22en-US%22%3E%3CP%3EI%20was%20recently%20asked%20to%20help%20out%20our%20friends%20at%20Channel%209%20with%20a%20storage%20problem.%20They%20capture%20large%20video%20files%20as%20part%20of%20their%20recordings%20and%20have%20to%20have%20them%20local%20for%20fast%20access%20in%20order%20to%20edit%20them.%20They%20currently%20use%20a%20proprietary%20Linux%20box%20for%20NFS%20storage%20which%20is%20just%20about%20full.%20I%20have%20spec'ed%20out%20a%20Storage%20Spaces%20Direct%20setup%20on%20Windows%202019%20from%20the%20%3CA%20href%3D%22https%3A%2F%2Fwww.microsoft.com%2Fcloud-platform%2Fazure-stack-hci-catalog%3FWT.mc_id%3Ditopstalk-blog-rclaus%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAzure%20Stack%20HCI%20catalog%3C%2FA%3E%20-%20but%20it%20won't%20be%20in%20for%20a%20while.%20This%20is%20the%20first%20bare%20metal%20system%20I've%20had%20to%20rack%20and%20stack%20in%20a%20while%20and%20it's%20part%20of%20a%20longer%20project%20-%20so%20I%20thought%20it%20would%20be%20good%20to%20write%20up%20a%20short%20set%20of%20topical%20blog%20posts%20to%20go%20along%20with%20the%20process.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20786px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F115566iDDF845F7F465D4F8%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22install-dedup-via-server-manager-1.png%22%20title%3D%22install-dedup-via-server-manager-1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-623521%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ERick%20Claus%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EWindows%20Server%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Microsoft

I was recently asked to help out our friends at Channel 9 with a storage problem. They capture large video files as part of their recordings and have to have them local for fast access in order to edit them. They currently use a proprietary Linux box for NFS storage which is just about full. I have spec'ed out a Storage Spaces Direct setup on Windows 2019 from the Azure Stack HCI catalog - but it won't be in for a while. This is the first bare metal system I've had to rack and stack in a while and it's part of a longer project - so I thought it would be good to write up a short set of topical blog posts to go along with the process.

 

Finding the right physical server from on-premises inventory

First off - I am spoiled by running a simple Azure command to provision whatever size box I need in a matter of minutes - this is going back in time for me. Luckily, I have an old DataOn Cluster-in-a-box system from a lab that has some decent (70 TB) worth of storage to temporarily hold them over until the new system arrives.

 

DataOnBox.JPG

 

Like a fresh coat of paint - flattening and rebuilding the servers from a USB media was kind of refreshing. In no time at all I have two nodes of a Windows Server 2019 up and running.

 

Installing Data Deduplication
The files they are storing includes large video files copied up to the system by various camera operators in the field. There isn't a sophisticated logging system on what file came where or if files have already been dumped up on the share. As a result - we've got duplicates. I figured one easy way of resolving this space issue is to enable deduplication. Data Deduplication in Windows Server 2019 is more efficient than previous releases - it can run in the background on multiple volumes with minimal impact on other workloads. If you want some more details on Deduplication - we've got a good article over on Microsoft Docs you should check out
The temporary server in question has the fresh install of Windows Server 2019 with all available drives configured in a storage pool with a single volume. I really just needed to get this thing up, enable deduplication and copy over the data from the Linux box until the new AzureStack HCI configuration comes in.
The fastest way to install the DataDeduplication feature is to crack open a PowerShell window and type in Add-WindowsFeature -name FS-Data-Deduplication

 

InstallDataDedup.png


Enabling and Configuring Data Deduplication on Data Drive
To enable the default settings for DataDeduplication for my E: drive- it's was simple as typing in Enable-DedupVolume E:

 

enableDedup.png

 

Because this is a temporary server - I want to configure the minimum file age to be "0" in order for the server to start deduplicating files on the next job schedule or manual kickoff. To change this parameter - type in Set-DedupVolume -Volume "E:" -MinimumFileAgeDays 0


Time to Finish Rack and Stack / Install

At this point we're all ready to go. It's a matter of transporting and installing this box into it's temporary home and transferring the files over to it from the Linux NAS box. But for this - you'll have to wait until the next post in this short series.


From start to finish - this took the better part of an afternoon to provision some bare metal, with 70 TB of storage as a temporary server. That is a tad longer then what seems like a very long 5 minutes to provision a system in Azure - but the proximity and fast access to these source files is what is important to the team at Channel 9.

 

All this being said - as I was going through this - I was thinking of all the value-add services from Azure I could bring to this on-prem bare metal box. Once the more permanent solution is in place - I'll go through the process of adding on a few of these Azure services to help me out: Azure File Sync, Azure Backup and a bunch of the Management add-ons to monitor the health of the system and so on.

 

What about you? Have you considered or started to explore any of the Hybrid Azure-to-on-prem services?

2 Comments
Occasional Contributor

Just curious, what's the data compression ratio in this case?

Microsoft

Hey @Kexy Biscuit  - the compression ratio will vary based on the type of files in use. I'll update the post once later this week once the data transfer and deduplication jobs have run with the production workload.  My "guestimate" is not that high for unique files as they are video files - but actual duplicate files due to user error or filing mishaps will be biggest winners.