Forum Discussion

mergene's avatar
mergene
Brass Contributor
Aug 08, 2020
Solved

Cross Workspace Analytic Rule Performance

Hi,

 

I have several workspaces containing SecurityEvent table having data. When I perform a union on the workspaces and then do the filtering, calculation, etc., the query consumes too much resources (which makes sense). example query:

union (workspace1.SecurityEvent, workspace2.SecurityEvent, workspace3.SecurityEvent)

| where EventID == "4624"

| where <other conditions>. 

 

To reduce the resource usage, I do a basic filtering before the union like below:

let workspace1_Events = workspace1.SecurityEvent | where EventID=="4624";

let workspace2_Events = workspace2.SecurityEvent | where EventID=="4624";

let workspace3_Events = workspace3.SecurityEvent | where EventID=="4624";

union (workspace1_Events, workspace2_Events, workspace3_Events)

| where <other conditions>

 

When running a query , I can use "search in" and it is more efficient than "union". However, I can't use this operator in the rule. Is there a way to write crossworkspace rule queries without having to duplicate the query for each workspace to make it more efficient? 

  • mergene 

     

    first, search should not be more performant than union, so the difference may be elsewhere. 

     

    As to your question: I can't think of a way to avoid repeating the query, however, I can make it simpler:

     

    let Filter = (T:(EventID:int))
    {
    T | where EventID == 4624
    };
    union (workspace("a").SecurityEvent | invoke Filter()),(workspace("b").SecurityEvent | invoke Filter())

     

    While for a single condition it might not be much shorter, it is still more elegant, and will become much shorter if you have more conditions.

  • Luizao_f's avatar
    Luizao_f
    Brass Contributor
    Good night, mergene
    I have a scenario similar to yours.
    I manage some clients via MSSP. I don't want to create the same rule on every client. For example, the "windows authentication failure" rule is the same for each client. How did you go about centralizing? Could you share what your experience is like in this regard?
  • mergene 

     

    first, search should not be more performant than union, so the difference may be elsewhere. 

     

    As to your question: I can't think of a way to avoid repeating the query, however, I can make it simpler:

     

    let Filter = (T:(EventID:int))
    {
    T | where EventID == 4624
    };
    union (workspace("a").SecurityEvent | invoke Filter()),(workspace("b").SecurityEvent | invoke Filter())

     

    While for a single condition it might not be much shorter, it is still more elegant, and will become much shorter if you have more conditions.

    • mergene's avatar
      mergene
      Brass Contributor

      Ofer_Shezaf I've asked the difference between "search in" and "union" in one of the MTP webinar and I was told that "search in" was more efficient (not the "search" itself). 

       

      Thanks for the suggestion. Is it possible to create complex filters having where, extend, summarize, and other operators with the method you suggested? Where can I find documentation about this?

      • Ofer_Shezaf's avatar
        Ofer_Shezaf
        Icon for Microsoft rankMicrosoft

        mergene 

         

        "search in" vs. "union": if you search in a small subset of tables and do "union *", then you might be faster. In general, search (and search in) is much slower: it usually implies "contains" across all fields which is the most expensive query. With union you can use any operator and more directly limit to fields.  

         

        Well, you can do search in (SecurityAlert, SecurityEvent) _ResourceId:"127.0.0.1", but then it would be exactly like a simlar union.

         

        As to my example: do whatever you like in the function. Just remember to declare any field from the source table you will reference. 

Resources