About defining DPM protection from CLI

[System-Center-Team]

First published on TECHNET on Oct 19, 2010

Those with interest in modifying and creating DPM protection from CLI probably appreciate this otherwise there is little sense in reading further I guess. Those that seek to save and recreate protection configurations are up for a task that proved to be a daunting one, you may want to look into a twin script called DPMsaveConfig / DPMcreateConfig .
We will assume a working knowledge of Powershell and involved commandlets as described here ; http://technet.microsoft.com/en-us/library/bb808941.aspx . This article supplements the aforementioned TechNet article for broader purposes than originally intended but is by no means complete in terms of all combinations and conditions you may encounter. Still, this covers a good deal of what you may want to know.

Required object instances

More complex tasks intended to be performed through the UI sometimes require objects and initialization normally provided by the UI under the hoods and therefore not documented as such. This also applies to parameter values that must fall within a particular range or fixed list of allowable values. Because we seek essentially an automated form of UI operations we have to mimic some functionality the UI uses under the hood.
How would you know when and how to do that? Well, continue reading… More details later but for now you need to understand that most object instances come from a preceding “Get-..” commandlet to be used in a subsequent “Set-…” and some have to be instantiated by yourself, for instance using;
$obj = New-Object -Typename <class name> [-Argumentlist arg1, arg2,…]

To be aware of

This section discusses some facts, tips & tricks you might appreciate.

One general caution is to not correlate wordings used in the UI with names used in the CLI. The UI is designed to be intuitive for common interpretation whilst the CLI is more geared towards technical implementation accuracy. In other words; a CLI parameter with a name also used in the UI may have a very different meaning or scope. Similarly, options in the UI may not even exist as parameter or property but implemented through multiple different combinations.

Order of executing protection commandlets

It may not be clear that commandlets involved with protection definitions must be used in a specific order because these cross influence each other. If you do not pay attention to this you may get confused by the same commandlet and parameter values returning different results or errors. You do not always have to use all of the below, just be aware of the ordering aspect;

(1) Set-ProtectionType: “how” like ‘DiskToDisk’ or ‘DiskToDiskToTape’ and so forth...
(2) Set-PolicyObjective: “goals” like recovery/retention range or in other words “how much”
(3) Set-PolicySchedule: “when” like daily at 08:00 or every 2 weeks and so forth…
(4) <other options>: like storage allocation, tape options and so forth…
(5) Set-ProtectionGroup: commits protection group changes to the DPM configuration

For instance, if protection type is set to ‘DiskToDisk’ you cannot use policy parameters that explicitly apply to long term protection only. Or if you change protection type from ‘DiskToDisk’ to include any long term protection the current objective and schedules may not be properly initialized. Although defaults apply, default interpretation may not always match your goal and may produce convoluting errors when executing the next commandlet or attempt to commit the change. This also means that a problem condition may not originate in the commandlet that reports it but from an earlier step, or omitted step and supplied defaults or current values may not fit what you are trying to accomplish.
Note : particularly with errors that report “Some prerequites not met…” should be read carefully because this often tells you which previous step was not done properly or which parameter does not fit into the current context . If there is no additional information on prerequisites you probably forgot to “Get-ModifiableProtectionGroup”.

About day and time parameter usage

Objectives and schedules deal with day and time specifications in various places that exist in singular and plural form with slight differences.
There is a ‘-TimeOfDay’ with values like “08:00 AM” but there is also a ‘-Time s OfDay’ which is an array of 1 or more “08:00 AM” like values. I recommend using 12 hour AM/PM notation format for self-documentary purposes. Similarly there is a ‘-DayOfWeek’ with values like “Monday” but also ‘-Day s OfWeek’ which is an array of short weekdays as in the note below.
Note : were the UI shows “every day” on the CLI level this specified as an array of all weekdays; “Mo”,”Tu”,”We”,Th”,”Fr”, “Sa”, Su”.

Long term interval frequency oriented parameter values come in a form like “Daily”, “Weekly”, “BiWeekly”, “Monthly” and so forth.
Short term intervals are specified in minutes from 1 to 1440 (24 hours).
Period oriented parameter values come in the form of a range/unit value collection like; 2,”weeks“ wherein range is “2” and unit is “weeks“ but note that for instance 2 years is specified as range “24” and unit “Months” (not 2, “Years” as you might anticipate).

A remarkable way of finding lacking information

I have no doubt there will be moments where you cannot find the information you need and like to share bit of trickery retrieving a required class and allowable parameter values. Not so spectacular is passing an object of the wrong type to a commandlet. The error message tells you something like “ cannot convert … of type-A into … of type-B” where type-B literally is the class name you need to typecast or use with “New-Object –Typename” that you also find in various blogged samples .
There is more fun with just specifying some parameter to an object that is not null and not a zero length string as value for which you have no clue what values are allowed. This will typically trigger an error message specifying possible values for the current context . This works best if you fully qualify all parameters by name like ‘–ProtectionGroup $pg’ and use objects of proper type but possibly incorrect value. One word of caution; in rare cases the CLI may decide to terminate this escapade. The file DPMCLI0curr.ERRLOG should provide a hint why and sometimes even lists allowable values. Here is a nice little sample excerpt of such (reformatted for readability);

01/14       19:36:32.386           69            WatsonIntegrator.cs(101)   -------------------
01/14       19:36:32.386           69            WatsonIntegrator.cs(107)  Expanding inner exceptions
01/14       19:36:32.386           69            WatsonIntegrator.cs(114)  --------------------
01/14       19:36:32.386           69            WatsonIntegrator.cs(114) Interval can only be 1 3, 6 or 12
01/14       19:36:32.386           69            WatsonIntegrator.cs(114)  -----------------
01/14       19:36:32.386           69            WatsonIntegrator.cs(114)
at Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.ArchiveSchedule.Validate
(DateTime startTime, Int32 interval, IntervalComponent component, Boolean isCustomScheme)

This exception occurred because I was playing with multiple custom schedules without first informing DPM that I wanted to go this way.

Let’s get crackin…

In this section the $mpg variable is assumed to be a modifiable group object obtained from New-ProtectionGroup or Get-ModifiableProtectionGroup. There are great many options but we limit to some common topics.

Notes on Set-ProtectionType

The following types can be selected. The ‘–ShortTerm’ and ‘–LongTerm’ parameters are shown separate for clarity but must be combined in 1 command if both protection schemes are desired.

• SetProtectionType –ProtectionGroup $mpg –ShortTerm “Disk”
• SetProtectionType –ProtectionGroup $mpg –ShortTerm “Media” (meaning tape)
• DPM2007: SetProtectionType –ProtectionGroup $mpg –LongTerm
• DPM2010: SetProtectionType –ProtectionGroup $mpg –LongTerm  “Tape” (required)
• <there some more for future use>
  
  

Notes on Set-PolicyObjective

-RetentionRangeInDays/inWeeks are short term parameters, note that these take an integer, not a retention range object. You combine this either with  ‘-SynchronizationFrequency’ in minutes or the switch ‘-BeforeRecoveryPoint’ but not both.

-RetentionRange is a long term parameter and takes a retention range object to be instantiated like below for a range of 3 and unit in months;

$RR= new-object -TypeName Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.RetentionRange -ArgumentList 3,"months”

You combine this with –LongtermBackupFrequency witch takes a string, for example;

Set-ProtectionObjective –ProtectionGroup $mpg –RetentionRange $RR –LongtermBackupFrequency “Weekly”

This would keep backup for 3 months and run weekly. Okay but when does that run?
That’s part of Set-PolicySchedule we talk about later.

-RetentionRangeList / -FrequencyList / -GenerationList
These are used to specify multiple objectives and are arrays referred to as custom schedules. We already talked about retention range and frequency types. The –GenerationList parameter may contain these values “GreatGrandFather”, “GrandFather”, “Father” and “Son” although ‘Son’ is used with the, for DPM unusual short term daily tape scenario.

Note : better tell DPM that we are going to play with custom schemes otherwise you may hit the exception as in the DPMUI0Curr.ERRLOG excerpt shown earlier;

$mpg.ArchiveIntent.IsCustomScheme = $true

So we need to populate these lists, say for 1 copy as is most usual, note oldest/longest first, in order and the same the across the triplets.

• Create the retention range list (GreatGrandFather, GrandFather, Father)

$rrGGF = New-Object -TypeName Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.RetentionRange -ArgumentList 12,"months"
$rrGF = New-Object -TypeName Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.RetentionRange –ArgumentList 3,"months"
$rrF= New-Object -TypeName Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.RetentionRange –ArgumentList 1,"months"
$RRlist = @($rrGGF,$rrGF,$rrF)

• Create the generation list

$GENlist = @(“GreatGrandFather”,”GrandFather”,”Father”)

• Create the frequency list

$FRQlist = @(12,3,1) #corresponds to ‘range’ property of recovery range object

We also need to initialize and assign a label information structure when using custom lists;

• Instantiate the label information object and assign values

$linfo = New-Object -TypeName Microsoft.Internal.EnterpriseStorage.Dls.XsdClasses.MM.Interface.labelinfo
$linfo.Vault = @("Offsite","Offsite","Offsite")                       # all 3 long term
$linfo.label=@("YearLabel","MonthLabel","Weeklabel") # just a sample format
$ linfo.Generation=@("GreatGrandFather","GrandFather","Father)

• Assign the mandatory protection group property and execute commandlet

$mpg.ArchiveIntent.LabelInfo=$linfo
Set-PolicyOBjective –Protectiongroup $mpg –RetentionRangeList $RRlist –FrequencyList $FRQlist –GenerationList $GENlist

That was for one copy, for two copies the lists and label information would have 6 elements. The order would be; first both GreatGrandFathers, then both GrandFathers and then both Fathers and all of the corresponding values in that order. Ensure both copy triplets have unique values for labels and vaults.

Notes on Set-PolicySchedule

Up to here we talked about how much we keep but not when, this we will talk about now.

Here too sets of parameters apply to short term or long term scenarios. Based on “Get-PolicySchedule” an object of desired type must be obtained and passed to the “Set-PolicySchedule” commandlet using modifying parameters. You typically obtain the object like (we’ll explain $jobtype later);

$sched = Get-PolicySchedule -ProtectionGroup $mpg -ShortTerm  |  where { $_.JobType -eq  $jobtype }
$sched = Get-PolicySchedule -ProtectionGroup $mpg -LongTerm  |  where { $_.JobType -eq  $jobtype }

What types ($jobtype) apply? The ones described below are those that relate to defining protection but there are more job types not shown here.

Short term

• “ShadowCopy” : this is a recovery point for file/volume protection
• “OnsiteSonArchive” : this is a disk to tape job
• “Replication” : synchronization job for volume or application incremental protection
• “FullReplicationForApplication” : this is an application ‘ExpressFull’ job
• “DiskAutoValidation” : this is a scheduled consistency check job

Long Term

• “OffsiteFatherArchive”, “OffsiteGrandFatherArchive”, “OffsiteGreatGrandFatherArchive”
  These are the 3 possible custom schedules for long term to tape protection schedules.

Now we are almost ready to set a policy schedule according our intentions, almost because we have to logically translate what our intentions are and how that maps to possible DPM operations. First think about the ‘type’ of operation and when that should happen rather than in terms of short and long term.

Take “ShadowCopy” and “FullReplicationForApplication” because those are similar. These can occur on 1 or more specific days or all and 1 or more times per day, in other words these are ‘plural’. Let’s assume we want to run these every day at 10:00 AM and 04:00 PM, that would be specified as;

$dow = @(Mo,Tu,We,Th,Fr,Sa,Su)          # just omit days you don’t want
$tod = @(10:00 AM, 04:00 PM)                # leave out or modify as you desire
Set-PolicySchedule -ProtectionGroup $mpg -Schedule $sched  -DaysOfWeek $dow –TimesOfDay $tod

A “Replication” job runs every so and so with an offset, say every 30 minutes or every 2 hours rather than a specific day and time. What can be a little confusing here is that synchronization is set as part of the objectives rather than in a schedule and leaves us to set the offset of say 3 minutes;

Set-PolicySchedule -ProtectionGroup $mpg -OffsetInMinutes 3

Archive jobs such as “OffsiteFatherArchive” run on a particular time and day, like Sundays at 11 PM;

Set-PolicySchedule -ProtectionGroup $mpg -Schedule $sched  -DayOfWeek “Sunday” –TimeOfDay “11:00 PM”

These can also have relative specifications like 1st day of each month etc…but I’ll let you play with that now you got the hang of it.

Notes on exclusions

So although we understand pretty much how to define protection objectives and schedules by now, there is this matter of excluding one or more folders or file types. Say you are protecting a volume but want to exclude MP3 and ISO files, this is how it is done;

Set-DatasourceProtectionoptions  $mpg  -FileType “MP3,ISO” –Add

The online help could be a little confusing with double negatives but just remember this; “-Add” creates more exclusions and less is protected, “-Remove” reduces exclusions and more gets protected. So if we have a whole bunch of types excluded and decide that we do want to protect ISO files and not exclude them from protection, we would do;

Set-DatasourceProtectionoptions  $mpg  -FileType “ISO” –Remove

So how do we exclude one or more folders from being protected, something similar? No, that would be too easy but we do stick to the ‘remove’ paradigm. We talk about data source hierarchy later but the idea is to remove a child from an already defined protection that normally includes the child. Forget about how we get the objects for a while and assume we have a protected “D:\” and $child object reflecting “D:\TEMP” we want to exclude. Then you would apply the exclusion as follows;

Remove-ChildDatasource $mpg –ChildDatasource $child

A data source is a top hierarchy below which ‘childs’ can exist. For volume protection childs are folders and files and obviously a child can in turn have childs of its own, deeper positioned folders and files. A volume root directory (like “D:\”) represents the entire volume. So how do we get objects for these to include or exclude? If you want to protect just one or few directories you prefer to ‘add’ just those rather than the entire volume and subsequently perform dozens if not hundreds or more ‘removes’, right? To complete the aforementioned exclude sample we first protect “D:\” and remove the single item we don’t want but note that we start at the parent (Get-Datasource) containing level and query that (Get-ChildDatasource). Subsequently we single out “D:\TEMP” to protect only.

$ps = Get-ProductionServer –DPMservername “DPMSERVER” | where {$_.ProductionServername –eq “PRODSERVER”}

$allds = Get-Datasource –ProductionServer $ps –Inquire


$parent = $allds | where {$_.LogicalPath –eq “D:\”}

Add-ChildDatasource –Protectiongroup $mpg –ChildDatasource $parent


$child = Get-ChildDatasource –ChildDatasource $parent -Inquire  | where {$_.Logicalpath –eq “D:\TEMP”}
Remove-ChildDatasource $mpg –ChildDatasource $child

If you only want to protect “D:\TEMP” in this case you would;

$ps = Get-ProductionServer –DPMservername “DPMSERVER” | where {$_.ProductionServername –eq “PRODSERVER”}


$allds = Get-Datasource –ProductionServer $ps –Inquire


$parent = $allds | where {$_.LogicalPath –eq “D:\”}

$child = Get-ChildDatasource –ChildDatasource $parent -Inquire  | where {$_.Logicalpath –eq “D:\TEMP”}


Add-ChildDatasource –Protectiongroup $mpg –ChildDatasource $child

Obviously not proficient with deep complex structures for which we should use DPM management shell search capabilities. That’s food for another blog on selectively working with data sources that also applies to recovery where it is much harder needed.

Notes on “Get-Datasource”

The ‘Get-Datasource’ commandlet has many formats for a specific purpose that are not interchangeable. The ‘-inquire’ switch queries the protected target and requires the target to be well connected like servers. The ‘-computernames’ switch is the opposite, it is intended for disconnected clients and just adds the target to the DPM configuration inheriting settings from the client protection group such as which directories to include and exclude and which file extensions to exclude (if any). Mark however that you could add a client data source as if it was a server (and reachable) but would not render disconnected client protection as most likely was the intend for client. Furthermore, if once adding a client data source as server protection you may have to  uninstall and re-install the agent on this client to be able to protect it as a disconnected-client data source.

• to ‘get’ datasources for server protection you have to use the “-inquire” switch.



• to ‘get’ datasources for client protection you have to use the following format;
  
  $ds = Get-Datasource -DpmServerName <dpm name> -ComputerNames <client name>
  
  
  
  
  

Notes on disk allocation and ‘–CalculateSize’

The last thing you do before committing protection using “Set-ProtectionGroup” is to ‘Get- and Set-DatasourceDiskallocation’. This section is by no means complete and just a few hints to overcome most common problems. So, what does this switch do? DPM initializes default disk allocations with “Get-DatasourceDiskAllocation” based on the volume size and the –CalculateSize switch tells DPM to look closer what is actually needed, for instance if you only protect one or few folders that require far less space than the entire volume. The general format is;

Get-DatasourceDiskallocation –Protectiongroup $mpg –Datasource $parent –CalculateSize


Set-DatasourceDiskallocation –Protectiongroup $mpg –Datsource $parent


Set-ReplicaCreationmethod –Protectiongroup $mpg –Manual


Set-Protectiongroup $mpg

The –CalculateSize switch is sometimes optional, sometimes mandatory and sometimes not allowed, nice isn’t it? First of all note that on DPM2007 this switch only applies to protection of volumes, files, folders and shares. On DPM2010 there can be other scenarios. Whether or not to use this switch depends on the scenario and are too many to walk through, instead we clarify some errors you may encounter;

“Set-DatasourceDiskallocation” may complain that prerequisite steps are not properly executed in which case first verify that the –Inquire switch was used with “Get-[Child]Datasource” on server protection and add or remove the –Calculate switch. If the switch is not allowed it is usually stated in the error and the reason why.

Note: on these errors the “Set-Protectiongroup” commit may still succeed but possibly using a larger disk allocation than needed.

“Set-Protectiongroup” may also complain about prerequisites and could apply to anything discussed in this blog not being done properly but start with the previous bullet. It may also report that disk allocation was not calculated and the “–calculate” switch can be added to server protection.

As general guideline; if you create a protection group from scratch and add a complete volume or application data source then do not use the –Calculate switch. When adding multiple folders or a folder on a volume a part of which is already protected then do use the –Calculate switch. Think about whether DPM has to create a new allocation or extend an existing allocation, in latter case you must use the –Calculate switch. As stated earlier, I am aware this does not cover all but the best I can do without going ballistic as far as this blog did not already.

If nothing else you do get an appreciation of what it takes to make a ‘simple’ backup product easy and intuitive to use when taken from the UI perspective, right?

Have fun!