Searching by more than one field when using a watch list

[ChristopherKerry]

Hi there,

I'm trying to filter by multiple fields in a watchlist. Something similar to the below, but with the fields user, src and dest.

| where SrcIP !in ((_GetWatchlist('WL_Global') | project user, src, dest | project-rename SrcIP=src))

I've tried the below, but this comes back with an error:

| where * !in ((_GetWatchlist('WL_Global') | project user, src, dest | project-rename SrcIP=src))

Is there a way of doing this without repeating the entire line for each separate field?

Many thanks

[#2]

So you will need a where statement regardless to filter on each column. If you want to filter multiple columns then you will need to have 3 where statements.

If you want to shorten the statement. You could save 3 queries as functions and just call the alias which will filter to 3 columns.
Then it would be
|where user in WL_Global_user |where src in WL_Global_src |where dest in WL_Global_dest

I don't recommend doing a wildcard (*) as it may be kind of slow. Is there a reason you are doing a wildcard. If the data you are doing the where statement is not normalized, it will be a very long calculation in my opinion.

Thoughts
Normalize that source data
No wildcards if that is possible.

[#3]

@TeachJing 

Thanks for the response TeachJing. The data is normalised already - it's more just to see if there was a more efficient way of doing things instead of having 3 where statements.

In splunk I would do something like this:
| search NOT [| inputlookup WL_Global | fields user src dest ] 

Splunk would translate that into optimised code that would then run on the indexers. I was wondering if there was an equivalent in Sentinel? Or is the best way with 3 where statements loading up the watchlist each time?

[#4]

tagging @CliveWatson, @Jeremy Tan and @Ofer_Shezaf in case they know of a better way and to document the way Splunk does it

[#5]

@ChristopherKerry 

This is what I really find challenging with Splunk. Queries are absolutely unreadable. The SPL example you brought is fast to write to the initiated but does make any sense logically, making it impossible to understand if you are not a Splunk Guru.

In KQL you have to be explicit, and readable, but I don't think makes the optimization different.

[#6]

@Ofer_Shezaf I would definitely say this is easier to read and quicker to write (especially when you have a lot of query to go through):

| search NOT [| inputlookup LOOKUP | fields src dest dest_port app protocol url]

than this:

| where SrcIP !in ((_GetWatchlist('LOOKUP') | project SrcIP))
| where Dest !in ((_GetWatchlist('LOOKUP') | project Dest))
| where DestPort !in ((_GetWatchlist('LOOKUP') | project DestPort))
| where App !in ((_GetWatchlist('LOOKUP') | project App))
| where Protocol !in ((_GetWatchlist('LOOKUP') | project Protocol))
| where Url !in ((_GetWatchlist('LOOKUP') | project Url))

Even more so when you have 3 or 4 lookups to correlate and you can end up with 10+ lines of KQL just for a few lookups.

Is there a better way to store this information and correlate it in Sentinel?

[#7]

@ChristopherKerry @Ofer_Shezaf 
To expand on this:

| search NOT [| inputlookup LOOKUP | fields src dest dest_port app

Splunk is parsing all the fields or columns you currently have available and matching them to the lookup, it then compares the value of that field and when it finds a match removes the result from our search.

I suppose a less verbose but similar approach in KQL land would be:

| where (SrcIP or User or Dest) !in ((_GetWatchlist('WL_Global') | project user, src, dest

Currently, the above doesn't work and you do have to individually split out your where filtering:

I can see both sides, but it's far more verbose in KQL making readability more of a struggle.

[#8]

Tagging @UriBarash and @Deepak Agrawal for visibility