Azure Monitor agent and Data Collection Rules

Published Oct 04 2021 11:06 PM 1,915 Views
Microsoft

Last week Sonia Cuff wrote about finding your Azure Log Analytics agent deployments in your environment in preparation for the Azure Monitor agent migration.

 

This was in response to an email you might have received if you are currently using the Log Analytics Agent.

 

We previously covered the reasons behind the need for a new agent.

 

 

But the reason for today’s article is to explore a bit the DCR or Data Collection Rules.

 

PierreRoman_0-1633413259693.png

 

DCRs are a way to define data coming into Azure Monitor and specify where that data should be sent or stored. And it opens possibilities.

 

Log Analytics Workspace Access

 

I’ve been asked before (many times) if it was possible for building hub-&-Spoke type Log Analytics workspace.

 

The reason that was given for the request was to allow a way to have a hierarchy in the access of logs and monitoring data.   Something like branch office admins would only have access to view, report and configure alerts & actions for data from their own branch.  Regional admins would have rights to all the branches data in their regions and finally Corp Admin would see all the data.

 

The way to achieve that used to be complicated, expensive (you would be charged for data ingestion in all workspaces) and not supported since it could be achieved in a much more elegant way by using the Resource-context access model in a single workspace instead of duplicating the data ingestion in multiple workspaces and using the Workspace-context access model.

 

Log Analytics workspace are containers that includes data and configuration information used by Azure Monitor.  And they serve as an administrative boundary.  There are 2 access control mode:

 

  • Workspace-context: You can view all logs in the workspace you have permission to
  • Resource-context: You can view logs for only resources in all tables that you have access to.

While you can deploy one or more workspaces in your Azure subscription, there are several considerations you should understand.  We covered this during our ITOpsTalks: All Things Hybrid event earlier this year.

 

 

 

How DCR can help.

If you have a real requirement for multiple workspaces based on one or more of the following requirements:

 

  • You are a global company, and you need log data stored in specific regions for data sovereignty or compliance reasons.
  • You are using Azure and you want to avoid outbound data transfer charges by having a workspace in the same region as the Azure resources it manages.
  • You manage multiple departments or business groups, and you want each to see their own data, but not data from others. Also, there is no business requirement for a consolidated cross department or business group view.

DCR could now be your key to having the data in multiple workspaces.

 

NOTE:  DCRs only collect data from virtual machines using the Azure Monitor agent

 

For example, a virtual machine may have an association to multiple DCRs.  This allows you to define a set of DCRs, each matching a particular requirement, and apply them to only the virtual machines where they apply.

 

For example, If you have sets of VMs running Line of business application (LOB) and other running SQL Server...

 

You could have one DCR that applies to all virtual machines and separate DCR that collect data specifically from the LOB VMs and for SQL Server.

 

PierreRoman_1-1633413425891.png

 

 

That way you could define in each DCRs where the data is being sent to, in effect sending the same data to multiple workspaces.  Keep in mind that your cost will increase since you are ingesting the same data in multiple workspaces.

 

Now,  if this is something you think can help your enterprise.  the first step is to migrate to the new Azure Monitor Agent.

Migration considerations

The Azure Monitor agent is generally available and fully supported, however it doesn’t yet have full feature parity with the Log Analytics agent. At the time of writing:

 

  • Not all Log Analytics solutions are supported today. Learn what's supported .
  • No support for Azure Private Links.
  • No support for collecting file based logs or IIS logs.

 

Defining the data collection rules is also handled differently in the Azure Monitor agent, allowing for more unique, scoped configurations for subsets of machines. Learn more at Changes in data collection.

 

Finally, see more migration considerations at Should I switch to the Azure Monitor agent?

 

I hope this helps.

 

Cheers!

 

Pierre

1 Comment
Contributor

So it his GA? They mentioned April/June of 2022 or this was recorded earlier and it's already GA?

%3CLINGO-SUB%20id%3D%22lingo-sub-2811603%22%20slang%3D%22en-US%22%3EAzure%20Monitor%20agent%20and%20Data%20Collection%20Rules%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2811603%22%20slang%3D%22en-US%22%3E%3CP%3ELast%20week%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F170596%3FWT.mc_id%3Dmodinfra-42040-pierrer%22%20target%3D%22_blank%22%3ESonia%20Cuff%3C%2FA%3E%20wrote%20about%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fitops-talk-blog%2Fhow-to-find-your-azure-log-analytics-agent-deployments-in%2Fba-p%2F2797887%3FWT.mc_id%3Dmodinfra-42040-pierrer%22%20target%3D%22_blank%22%3Efinding%20your%20Azure%20Log%20Analytics%20agent%20deployments%3C%2FA%3E%20in%20your%20environment%20in%20preparation%20for%20the%20Azure%20Monitor%20agent%20migration.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20was%20in%20response%20to%20an%20email%20you%20might%20have%20received%20if%20you%20are%20currently%20using%20the%20Log%20Analytics%20Agent.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20previously%20covered%20the%20reasons%20behind%20the%20need%20for%20a%20new%20agent.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CIFRAME%20src%3D%22%20https%3A%2F%2Fchannel9.msdn.com%2FShows%2FIT-Ops-Talk%2FITOpsTalk-Azure-Monitor-Agent%2Fplayer%3FWT.mc_id%3Dmodinfra-42040-pierrer%22%20width%3D%22960%22%20height%3D%22540%22%20frameborder%3D%220%22%20allowfullscreen%3D%22allowfullscreen%22%20title%3D%22ITOpsTalk%3A%20Azure%20Monitor%20Agent%20-%20Microsoft%20Channel%209%20Video%22%3E%3C%2FIFRAME%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EBut%20the%20reason%20for%20today%E2%80%99s%20article%20is%20to%20explore%20a%20bit%20the%20DCR%20or%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fazure-monitor%2Fagents%2Fdata-collection-rule-overview%3FWT.mc_id%3Dmodinfra-42040-pierrer%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EData%20Collection%20Rules%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22PierreRoman_0-1633413259693.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F315158iD4BA1950C36B6324%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22PierreRoman_0-1633413259693.png%22%20alt%3D%22PierreRoman_0-1633413259693.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EDCRs%20are%20a%20way%20to%20define%20data%20coming%20into%20Azure%20Monitor%20and%20specify%20where%20that%20data%20should%20be%20sent%20or%20stored.%20And%20it%20opens%20possibilities.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--327599565%22%20id%3D%22toc-hId--327599537%22%20id%3D%22toc-hId--327599537%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH2%20id%3D%22toc-hId--2135054028%22%20id%3D%22toc-hId--2135054000%22%20id%3D%22toc-hId--2135054000%22%3ELog%20Analytics%20Workspace%20Access%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%E2%80%99ve%20been%20asked%20before%20(many%20times)%20if%20it%20was%20possible%20for%20building%20hub-%26amp%3B-Spoke%20type%20Log%20Analytics%20workspace.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20reason%20that%20was%20given%20for%20the%20request%20was%20to%20allow%20a%20way%20to%20have%20a%20hierarchy%20in%20the%20access%20of%20logs%20and%20monitoring%20data.%20%26nbsp%3B%26nbsp%3BSomething%20like%20branch%20office%20admins%20would%20only%20have%20access%20to%20view%2C%20report%20and%20configure%20alerts%20%26amp%3B%20actions%20for%20data%20from%20their%20own%20branch.%20%26nbsp%3BRegional%20admins%20would%20have%20rights%20to%20all%20the%20branches%20data%20in%20their%20regions%20and%20finally%20Corp%20Admin%20would%20see%20all%20the%20data.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20way%20to%20achieve%20that%20used%20to%20be%20complicated%2C%20expensive%20(you%20would%20be%20charged%20for%20data%20ingestion%20in%20all%20workspaces)%20and%20not%20supported%20since%20it%20could%20be%20achieved%20in%20a%20much%20more%20elegant%20way%20by%20using%20the%20%3CSTRONG%3EResource-context%3C%2FSTRONG%3E%20access%20model%20in%20a%20single%20workspace%20instead%20of%20duplicating%20the%20data%20ingestion%20in%20multiple%20workspaces%20and%20using%20the%20%3CSTRONG%3EWorkspace-context%20%3C%2FSTRONG%3Eaccess%20model.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ELog%20Analytics%20workspace%20are%20containers%20that%20includes%20data%20and%20configuration%20information%20used%20by%20Azure%20Monitor.%20%26nbsp%3BAnd%20they%20serve%20as%20an%20administrative%20boundary.%26nbsp%3B%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fazure-monitor%2Flogs%2Fdesign-logs-deployment%3FWT.mc_id%3Dmodinfra-42040-pierrer%23access-mode%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EThere%20are%202%20access%20control%20mode%3C%2FA%3E%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CSTRONG%3EWorkspace-context%3C%2FSTRONG%3E%3A%20You%20can%20view%20all%20logs%20in%20the%20workspace%20you%20have%20permission%20to%3C%2FLI%3E%0A%3CLI%3E%3CSTRONG%3EResource-context%3A%20%3C%2FSTRONG%3EYou%20can%20view%20logs%20for%20only%20resources%20in%20all%20tables%20that%20you%20have%20access%20to.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EWhile%20you%20can%20deploy%20one%20or%20more%20workspaces%20in%20your%20Azure%20subscription%2C%20there%20are%20several%20considerations%20you%20should%20understand.%26nbsp%3B%20We%20covered%20this%20during%20our%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fitops-talk-blog%2Fbg-p%2FITOpsTalkBlog%2Flabel-name%2FITOps%2520Talks%3FWT.mc_modinfra-12531-socuff%26amp%3BWT.mc_id%3Dmodinfra-42040-pierrer%22%20target%3D%22_blank%22%3EITOpsTalks%3A%20All%20Things%20Hybrid%3C%2FA%3E%20event%20earlier%20this%20year.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CIFRAME%20src%3D%22https%3A%2F%2Fchannel9.msdn.com%2FShows%2FIT-Ops-Talk%2FOPS115-Log-Analytics-workspace-design-deep-dive%2Fplayer%3FWT.mc_id%3Dmodinfra-42040-pierrer%22%20width%3D%22960%22%20height%3D%22540%22%20frameborder%3D%220%22%20allowfullscreen%3D%22allowfullscreen%22%20title%3D%22OPS115%20-%20Log%20Analytics%20workspace%20design%20deep%20dive%20-%20Microsoft%20Channel%209%20Video%22%3E%3C%2FIFRAME%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-352458805%22%20id%3D%22toc-hId-352458833%22%20id%3D%22toc-hId-352458833%22%3EHow%20DCR%20can%20help.%3C%2FH2%3E%0A%3CP%3EIf%20you%20have%20a%20real%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fazure-monitor%2Flogs%2Fdesign-logs-deployment%3FWT.mc_id%3Dmodinfra-42040-pierrer%23important-considerations-for-an-access-control-strategy%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Erequirement%20for%20multiple%20workspaces%3C%2FA%3E%20based%20on%20one%20or%20more%20of%20the%20following%20requirements%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EYou%20are%20a%20global%20company%2C%20and%20you%20need%20log%20data%20stored%20in%20specific%20regions%20for%20data%20sovereignty%20or%20compliance%20reasons.%3C%2FLI%3E%0A%3CLI%3EYou%20are%20using%20Azure%20and%20you%20want%20to%20avoid%20outbound%20data%20transfer%20charges%20by%20having%20a%20workspace%20in%20the%20same%20region%20as%20the%20Azure%20resources%20it%20manages.%3C%2FLI%3E%0A%3CLI%3EYou%20manage%20multiple%20departments%20or%20business%20groups%2C%20and%20you%20want%20each%20to%20see%20their%20own%20data%2C%20but%20not%20data%20from%20others.%20Also%2C%20there%20is%20no%20business%20requirement%20for%20a%20consolidated%20cross%20department%20or%20business%20group%20view.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EDCR%20could%20now%20be%20your%20key%20to%20having%20the%20data%20in%20multiple%20workspaces.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3ENOTE%3A%26nbsp%3B%20DCRs%20only%20collect%20data%20from%20virtual%20machines%20using%20the%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fazure-monitor%2Fagents%2Fazure-monitor-agent-migration%3FWT.mc_id%3Dmodinfra-42040-pierrer%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EAzure%20Monitor%20agent%3C%2FA%3E%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20example%2C%20a%20virtual%20machine%20may%20have%20an%20association%20to%20multiple%20DCRs.%26nbsp%3B%20This%20allows%20you%20to%20define%20a%20set%20of%20DCRs%2C%20each%20matching%20a%20particular%20requirement%2C%20and%20apply%20them%20to%20only%20the%20virtual%20machines%20where%20they%20apply.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20example%2C%20If%20you%20have%20sets%20of%20VMs%20running%20Line%20of%20business%20application%20(LOB)%20and%20other%20running%20SQL%20Server...%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20could%20have%20one%20DCR%20that%20applies%20to%20all%20virtual%20machines%20and%20separate%20DCR%20that%20collect%20data%20specifically%20from%20the%20LOB%20VMs%20and%20for%20SQL%20Server.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22PierreRoman_1-1633413425891.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F315159iBA0F0E6CDE301A53%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22PierreRoman_1-1633413425891.png%22%20alt%3D%22PierreRoman_1-1633413425891.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThat%20way%20you%20could%20define%20in%20each%20DCRs%20where%20the%20data%20is%20being%20sent%20to%2C%20in%20effect%20sending%20the%20same%20data%20to%20multiple%20workspaces.%26nbsp%3B%20Keep%20in%20mind%20that%20your%20cost%20will%20increase%20since%20you%20are%20ingesting%20the%20same%20data%20in%20multiple%20workspaces.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENow%2C%26nbsp%3B%20if%20this%20is%20something%20you%20think%20can%20help%20your%20enterprise.%26nbsp%3B%20the%20first%20step%20is%20to%20migrate%20to%20the%20new%20Azure%20Monitor%20Agent.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--1476055721%22%20id%3D%22toc-hId--1454995658%22%20id%3D%22toc-hId--1454995630%22%20id%3D%22toc-hId--1454995630%22%3EMigration%20considerations%3C%2FH2%3E%0A%3CP%3EThe%20Azure%20Monitor%20agent%20is%20generally%20available%20and%20fully%20supported%2C%20however%20it%20doesn%E2%80%99t%20yet%20have%20full%20feature%20parity%20with%20the%20Log%20Analytics%20agent.%20At%20the%20time%20of%20writing%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3ENot%20all%20Log%20Analytics%20solutions%20are%20supported%20today.%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fazure-monitor%2Fagents%2Fazure-monitor-agent-overview%3FWT.mc_id%3Dmodinfra-42040-pierrer%23supported-services-and-features%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ELearn%20what's%20supported%3C%2FA%3E%26nbsp%3B.%3C%2FLI%3E%0A%3CLI%3ENo%20support%20for%20Azure%20Private%20Links.%3C%2FLI%3E%0A%3CLI%3ENo%20support%20for%20collecting%20file%20based%20logs%20or%20IIS%20logs.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EDefining%20the%20data%20collection%20rules%20is%20also%20handled%20differently%20in%20the%20Azure%20Monitor%20agent%2C%20allowing%20for%20more%20unique%2C%20scoped%20configurations%20for%20subsets%20of%20machines.%20Learn%20more%20at%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fazure-monitor%2Fagents%2Fazure-monitor-agent-overview%3FWT.mc_id%3Dmodinfra-42040-pierrer%23changes-in-data-collection%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EChanges%20in%20data%20collection.%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFinally%2C%20see%20more%20migration%20considerations%20at%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fazure-monitor%2Fagents%2Fazure-monitor-agent-overview%3FWT.mc_id%3Dmodinfra-42040-pierrer%26amp%3Btabs%3DPowerShellWindows%23should-i-switch-to-the-azure-monitor-agent%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EShould%20I%20switch%20to%20the%20Azure%20Monitor%20agent%3F%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20hope%20this%20helps.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECheers!%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EPierre%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2811603%22%20slang%3D%22en-US%22%3E%3CP%3EIn%20this%20article%20we%20re-visit%20old%20questions%20that%20now%20have%20new%20answers%20due%20to%20the%26nbsp%3BAzure%20Monitor%20agent%20and%20Data%20Collection%20Rules.%20We'll%20cover%20access%20models%20and%20possibilities%20based%20on%20your%20requirements.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ERead%20on!%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2811603%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EData%20Collection%20Rules%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EPierre%20Roman%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2813649%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Monitor%20agent%20and%20Data%20Collection%20Rules%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2813649%22%20slang%3D%22en-US%22%3E%3CP%3ESo%20it%20his%20GA%3F%20They%20mentioned%20April%2FJune%20of%202022%20or%20this%20was%20recorded%20earlier%20and%20it's%20already%20GA%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E
Co-Authors
Version history
Last update:
‎Oct 04 2021 11:06 PM
Updated by: