In every OM12 Management Group, there is a “APM event collection Rule”. One and only one. What this does it, for each application configured for APM, subscribes to the APM Dispatcher (part of the “System Center Management APM” service) and pulls ALL events coming from the interceptor (=our instrumentation loaded into the monitored app) and sends them up to a Collection Management Server to store in the database. This makes the event available for access thru AppDiagnostics. Both the data source to collect events and the Write Action that stores them are code that is present on the agent (some installed with it, some delivered thru the dependency download mechanism = i.e. the module’s DLLs are bundled with Management Packs).
There are other rules that optionally alert on those events (or their quantity), but this is another story - we talked about them here already .
So back to our “APM event collection Rule”, then. Here it is, in case you were searching for it and got lost:
As I already wrote, what this rule does is to just collect ALL events produced by the APM agent. As simple as that.
This obviously works. Keep it simple.
Well… you know, it turns out that in some cases maybe we made this too simple. Let me give you an example.
Take the case of an agent that is multi-homed, for example, and the machine hosts three applications, say A, B and C. Say that, from the first management group, I have configured APM for application A and B. The moment that configuration becomes active, the second management group also discovers and starts considering that agent “activated” for APM monitoring. It therefore starts executing that rules.
Did you see what just happened?
Right: the second management group now is ALSO receiving and processing and storing ALL events coming from application A and B. There is an impact to performance, especially network and database (APM events are big in size, with all their details!).
Well, given that our scale numbers are smaller with APM than they are without it, you would think you would keep your APM data only reporting to one Management Group, and just disable the rule with an override on the second MG. Of course that would solve the issue.
Here I am going to present a more advanced/complex way to implement actual filtering, so that you can create a custom rule that has awareness of application names and only collect events for the applications you care about, only where you care about.
The built-in rule has three modules and looks like the following:
As we said earlier, DS and WA are custom code. CD is a standard composite module like many filters in other MPs.
For our new rule we need the same building blocks, some of which will need to be slightly modified.
- Data Source is easy – we can just use the same datasource that the rule provided with OM uses: Microsoft.SystemCenter.Apm.LobDatasource , defined in Microsoft.SystemCenter.Apm.Infrastructure management pack. What this datasource gives us access to, is the stream of events coming from the Avicode dispatcher living in the “System Center Management APM” Service (APM Service in short). Refer to my architectural diagram in my other blog post here http://blogs.technet.com/b/momteam/archive/2011/08/12/application-performance-monitoring-in-op... to see this “DS” and the APM Service on the agent. For this module, and for other things, our management pack needs a reference to the Microsoft.SystemCenter.Apm.Infrastructure management pack (we’ll call it MSAI!)
The Condition Detection and the Write Action are a bit more involved.
- The Write Action that is responsible for actually writing the data in the database is the right APM tables is marked as “internal” accessibility in the original management pack that ships with OM… which means we cannot directly reference it. In order to use the functionality, albeit we aren’t modifying anything, we essentially need to duplicate it in our MP. Not very hard, here it is:
<WriteActionModuleType ID="Microsoft.SystemCenter.Apm.CollectApmEventWriteAction.Custom" Batching="true" Accessibility="Internal" RunAs="SC!Microsoft.SystemCenter.DatabaseWriteActionAccount" Comment="Stores events into the Operations Manager database">
<Configuration></Configuration>
< ModuleImplementation Isolation="OwnProcess">
<Managed>
<Assembly>MSAI!Res.Microsoft.SystemCenter.Apm.ServerModules</Assembly>
<Type>Microsoft.EnterpriseManagement.Modules.Apm.ServerModules.SEViewerWriteAction</Type>
</Managed>
</ModuleImplementation>
< InputType>System!System.BaseData</InputType>
</WriteActionModuleType>
</ModuleTypes>
< /TypeDefinitions>
- The Condition Detection is the most interesting part: this is the piece that actually looks at the incoming event and decides whether it is “interesting” for the rule to actually pass it along to the WA, and eventually have it stored in the database. If the condition detection filter doesn’t consider the incoming data item as “interesting” (=matching its criteria) then the event will essentially be lost and not transported out of the agent nor stored.
This CD is actually a composite module which includes two condition detections:
- one is the actual filter - THIS IS THE PIECE WE NEED TO CHANGE in order to select items from specific applications only.
- the second one is a module we ship that applies aliasing information (i.e. transforms the incoming event and adds additional information to it)
I highlighted the filter I provided as sample, below. Basically, all you would have to do is to change this expression to match what you want it to match. It is standard management packs’ expression syntax, so – if you have been using OM in the past – this should be nothing new:
<ConditionDetectionModuleType ID="Microsoft.SystemCenter.Apm.ApplicationFilter" Batching="false" PassThrough="true" Stateful="false" Accessibility="Internal" Comment="Filters for AVICODE perf counter events">
<Configuration />
< ModuleImplementation>
<Composite>
< MemberModules>
< ConditionDetection ID="CDFilterApplication" TypeID="System!System.ExpressionFilter">
<Expression>
< SimpleExpression>
< ValueExpression>
< XPathQuery Type="String">/DataItem/eventSource</XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
< ValueExpression>
<Value Type="String">DinnerNow - Production</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</ConditionDetection>
< ConditionDetection ID="Aliasing" TypeID="MSAI!Microsoft.SystemCenter.Apm.AliasingHiddenRulesTransform"></ConditionDetection>
</MemberModules>
<Composition>
<Node ID="CDFilterApplication">
<Node ID="Aliasing" />
</Node>
</Composition>
</Composite>
</ModuleImplementation>
< OutputType>System!System.BaseData</OutputType>
< InputTypes>
< InputType>System!System.BaseData</InputType>
</InputTypes>
< /ConditionDetectionModuleType>
And then once the above set of modules is in place, this is how the actual custom rule looks like, after plumbing those modules together:
<Rule ID="Microsoft.SystemCenter.Apm.CollectApplicationDiagnosticsEvents" Comment="Collect all APM events" Enabled="true" Target="MSAI!Microsoft.SystemCenter.Apm.ApmAgent" ConfirmDelivery="true" Remotable="false" Priority="Normal" DiscardLevel="100">
<Category>Custom</Category>
< DataSources>
< DataSource ID="AVIDatasource" TypeID="MSAI!Microsoft.SystemCenter.Apm.LobDatasource" />
</DataSources>
< ConditionDetection ID="FilterAndAliasing" TypeID="Microsoft.SystemCenter.Apm.ApplicationFilter" />
< WriteActions>
< WriteAction ID="ApmWriteAction" TypeID="Microsoft.SystemCenter.Apm.CollectApmEventWriteAction.Custom" Target="SC!Microsoft.SystemCenter.CollectionManagementServer" />
</WriteActions>
< /Rule>
Those are the key pieces you need in an MP to take advantage of advanced filtering.
Finally, if you only import this new rule BUT also leave the default one running, you will not see any immediate difference, if you also don’t disable the original rule… so, in order to plug in our own custom rule as the only one, we need to disable the original one with an override:
<RulePropertyOverride ID="OverrideForRuleMicrosoftSystemCenterApmCollectApplicationDiagnosticsEventsForContextMicrosoftSystemCenterApmActivatedApmAgente9dbb82086aa460a8c280513917e1a26" Context="MSAI!Microsoft.SystemCenter.Apm.ApmAgent" Enforced="false" Rule="MSAI!Microsoft.SystemCenter.Apm.CollectApplicationDiagnosticsEvents" Property="Enabled">
<Value>false</Value>
< /RulePropertyOverride>
The full MP is attached. Use wisely.
Disclaimer
This posting is provided "AS IS" with no warranties, and confers no rights. Use of included utilities are subject to the terms specified at http://www.microsoft.com/info/copyright.htm