Cross Workspace Analytic Rule Performance

[mergene]

Hi,

I have several workspaces containing SecurityEvent table having data. When I perform a union on the workspaces and then do the filtering, calculation, etc., the query consumes too much resources (which makes sense). example query:

union (workspace1.SecurityEvent, workspace2.SecurityEvent, workspace3.SecurityEvent)

| where EventID == "4624"

| where <other conditions>. 

To reduce the resource usage, I do a basic filtering before the union like below:

let workspace1_Events = workspace1.SecurityEvent | where EventID=="4624";

let workspace2_Events = workspace2.SecurityEvent | where EventID=="4624";

let workspace3_Events = workspace3.SecurityEvent | where EventID=="4624";

union (workspace1_Events, workspace2_Events, workspace3_Events)

| where <other conditions>

When running a query , I can use "search in" and it is more efficient than "union". However, I can't use this operator in the rule. Is there a way to write crossworkspace rule queries without having to duplicate the query for each workspace to make it more efficient? 

[#2 · Accepted solution]

@mergene 

first, search should not be more performant than union, so the difference may be elsewhere. 

As to your question: I can't think of a way to avoid repeating the query, however, I can make it simpler:

let Filter = (T:(EventID:int))
{
 T | where EventID == 4624
};
union (workspace("a").SecurityEvent | invoke Filter()),(workspace("b").SecurityEvent | invoke Filter()) 

While for a single condition it might not be much shorter, it is still more elegant, and will become much shorter if you have more conditions.

[#3]

@Ofer_Shezaf I've asked the difference between "search in" and "union" in one of the MTP webinar and I was told that "search in" was more efficient (not the "search" itself). 

Thanks for the suggestion. Is it possible to create complex filters having where, extend, summarize, and other operators with the method you suggested? Where can I find documentation about this?

[#4]

@mergene 

"search in" vs. "union": if you search in a small subset of tables and do "union *", then you might be faster. In general, search (and search in) is much slower: it usually implies "contains" across all fields which is the most expensive query. With union you can use any operator and more directly limit to fields.  

Well, you can do search in (SecurityAlert, SecurityEvent) _ResourceId:"127.0.0.1", but then it would be exactly like a simlar union.

As to my example: do whatever you like in the function. Just remember to declare any field from the source table you will reference. 

[#5]

Thanks Ofer! Creating a function and doint an iteration looks very useful.

[#6]

Good night, @mergene
I have a scenario similar to yours.
I manage some clients via MSSP. I don't want to create the same rule on every client. For example, the "windows authentication failure" rule is the same for each client. How did you go about centralizing? Could you share what your experience is like in this regard?

[#7]

@Luizao_f 

You can transform a query into a function like in this post. Then invoke the function in one rule like:

union ( workspace01| invoke function_x()), (workspace02| invoke function_x()) )

There might be better ways. There is a MSSP guide for Azure Sentinel that you can check, if not already. 

[#8]

Also see https://techcommunity.microsoft.com/t5/azure-sentinel/what-s-new-cross-workspace-analytics-rules/ba-...