SOLVED

Watchlist and query

%3CLINGO-SUB%20id%3D%22lingo-sub-1834865%22%20slang%3D%22en-US%22%3EWatchlist%20and%20query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1834865%22%20slang%3D%22en-US%22%3E%3CP%3Enew%20to%20kql%20here%2C%20is%20it%20possible%20to%20build%20a%26nbsp%3B%20query%20that%20search's%20across%20logs%20looking%20for%20machines%20that%20connected%20to%20any%20of%20ip%20addresses%20in%20the%20watchlist%3F%26nbsp%3B%20Any%20examples%20%3F%20Plan%20would%20be%20to%20turn%20that%20query%20into%20a%20log%20analytic%20rule%20to%20create%20events%20and%20eventually%20a%20playbook.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Ethanks%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1837178%22%20slang%3D%22en-US%22%3ERe%3A%20Watchlist%20and%20query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1837178%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F850945%22%20target%3D%22_blank%22%3E%40roadruner%3C%2FA%3E%26nbsp%3BHere%20is%20a%20simple%20example%20of%20how%20to%20do%20this.%26nbsp%3B%20I%20created%20a%20CSV%20file%20that%20has%20all%20the%20IPAddresses%20I%20have%20cleared%20and%20uploaded%20that%20into%20the%20Watchlist%20using%20%22ClearedIPAddreses%22%20as%20the%20alias.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CDIV%3E%3CDIV%3E%3CSPAN%3Elet%20ClearedIPAddresses%3D_GetWatchlist(%3C%2FSPAN%3E%3CSPAN%3E'ClearedIPAddresses'%3C%2FSPAN%3E%3CSPAN%3E)%3B%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%3E%3CSPAN%3EHeartbeat%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%3E%3CSPAN%3E%7C%20%3C%2FSPAN%3E%3CSPAN%3Ejoin%3C%2FSPAN%3E%3CSPAN%3E%20ClearedIPAddresses%20%3C%2FSPAN%3E%3CSPAN%3Eon%3C%2FSPAN%3E%20%3CSPAN%3E%24%3C%2FSPAN%3E%3CSPAN%3Eleft.ComputerIP%20%3D%3D%20%3C%2FSPAN%3E%3CSPAN%3E%24%3C%2FSPAN%3E%3CSPAN%3Eright.IPAddress%3C%2FSPAN%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1837504%22%20slang%3D%22en-US%22%3ERe%3A%20Watchlist%20and%20query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1837504%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F46875%22%20target%3D%22_blank%22%3E%40Gary%20Bushey%3C%2FA%3E%26nbsp%3B%20Hi%2C%20i%20tried%20that%20query%20with%20alias%20of%20test1%20which%20is%20alias%20of%20watchlist%20and%20received%20an%20error%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CDIV%3E%3CSPAN%3Elet%20ClearedIPAddresses%3D_GetWatchlist(%3C%2FSPAN%3E%3CSPAN%3E'test1'%3C%2FSPAN%3E%3CSPAN%3E)%3B%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%3E%3CSPAN%3EHeartbeat%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%3E%3CSPAN%3E%7C%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3Ejoin%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3BClearedIPAddresses%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3Eon%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%24%3C%2FSPAN%3E%3CSPAN%3Eleft.ComputerIP%20%3D%3D%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E%24%3C%2FSPAN%3E%3CSPAN%3Eright.IPAddress%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%3E%3CSPAN%3Eerror%20is%20'join'%20operator%3A%20failed%20to%20resolve%20Column%26nbsp%3Bnamed%20%22IPAddress%22%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%3E%3CSPAN%3Emy%20csv%20file%20has%20the%20name%20IP%20Addresses%20in%20first%20cell%20then%20next%20cells%20below%20the%20actual%20ip%20addresses.%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%3E%3CSPAN%3EWhat%20do%20you%20mean%20by%20cleared%3F%20The%20ip's%20I%20would%20have%20in%20my%20list%20would%20be%20IOC's%2C%20thus%20checking%20to%20see%20if%20any%20machines%20were%20hitting%20them.%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%3E%3CSPAN%3Ethanks%20again%3C%2FSPAN%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1837978%22%20slang%3D%22en-US%22%3ERe%3A%20Watchlist%20and%20query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1837978%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F850945%22%20target%3D%22_blank%22%3E%40roadruner%3C%2FA%3E%26nbsp%3BYou%20CSV%20file%20would%20need%20to%20have%20the%20column%20headers%20in%20the%20first%20row.%26nbsp%3B%20One%20of%20mine%20was%20%22IPAddresses%22%2C%20you%20would%20need%20to%20substitute%20whatever%20you%20called%20your%20columns%20for%20that.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20used%20the%20term%20%22cleared%22%20only%20because%20my%20watchlist%20contained%20those%20IP%20Addresses%20that%20I%20want%20to%20allow.%26nbsp%3B%20You%20can%20call%20you%20watchlist%20whatever%20makes%20sense%20to%20you.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1838596%22%20slang%3D%22en-US%22%3ERe%3A%20Watchlist%20and%20query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1838596%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F46875%22%20target%3D%22_blank%22%3E%40Gary%20Bushey%3C%2FA%3E%26nbsp%3BThanks%20I%20tracked%20the%20error%20I%20had%2C%20which%20was%20the%20columns.%20It%20runs%20with%20no%20errors%20now.%20I%20did%20run%20a%20quick%20test%20and%20hit%20one%20of%20the%20ip's%26nbsp%3B%20in%20watchlist%20and%20then%20ran%20the%20query%20and%20no%20results%20found.%20Does%20the%20query%20search%20out%20all%20of%20sentinel%3F%20I%20tried%20just%20putting%20CommonSecurityLog%20to%20see%20if%20it%20would%20just%20search%20through%20those%20logs%2C%20since%20that's%20where%20the%20hit%20should%20be.%3C%2FP%3E%3CP%3EHere%20is%20what%20i%20tried%2C%20didn't%20work.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ECommonSecurityLog%3C%2FP%3E%3CP%3Elet%20ClearedIPAddresses%3D_GetWatchlist('test1')%3B%3CBR%20%2F%3EHeartbeat%3CBR%20%2F%3E%7C%20join%20ClearedIPAddresses%20on%20%24left.ComputerIP%20%3D%3D%20%24right.IPAddresses%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1838663%22%20slang%3D%22en-US%22%3ERe%3A%20Watchlist%20and%20query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1838663%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F850945%22%20target%3D%22_blank%22%3E%40roadruner%3C%2FA%3E%26nbsp%3BIt%20will%20only%20search%20the%20one%20table.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThere%20really%20is%20no%20way%20to%20search%20all%20tables%20for%20multiple%20values.%26nbsp%3B%20There%20is%20the%20%22search%22%20command%2C%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-explorer%2Fkusto%2Fquery%2Fsearchoperator%2C%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-explorer%2Fkusto%2Fquery%2Fsearchoperator%2C%3C%2FA%3E%26nbsp%3Bthat%20allows%20you%20to%20search%20for%20a%20single%20term%20although%20I%20do%20not%20know%20if%20this%20can%20be%20used%20in%20a%20Analytic%20rule%20or%20not.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1839232%22%20slang%3D%22en-US%22%3ERe%3A%20Watchlist%20and%20query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1839232%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F46875%22%20target%3D%22_blank%22%3E%40Gary%20Bushey%3C%2FA%3E%26nbsp%3BHmm%20ok%2C%20thanks.%26nbsp%3B%20How%20can%20i%20search%20one%20table%3F%20say%20CommonSecurityLog%3C%2FP%3E%3CP%3EI%20tried%20this%20but%20no%20dice.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ECommonSecurityLog%3C%2FP%3E%3CP%3Elet%20ClearedIPAddresses%3D_GetWatchlist('test1')%3B%3CBR%20%2F%3EHeartbeat%3CBR%20%2F%3E%7C%20join%20ClearedIPAddresses%20on%20%24left.ComputerIP%20%3D%3D%20%24right.IPAddress%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1839649%22%20slang%3D%22en-US%22%3ERe%3A%20Watchlist%20and%20query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1839649%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F850945%22%20target%3D%22_blank%22%3E%40roadruner%3C%2FA%3E%26nbsp%3BThis%20is%20the%20starting%20query%20for%20something%20like%20that.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Elet%20ClearedIPAddresses%3D_GetWatchlist('test1')%3B%3CBR%20%2F%3ECommonSecurityLog%3CBR%20%2F%3E%7C%20join%20ClearedIPAddresses%20on%20%24left.SourceIP%3D%3D%20%24right.IPAddress%3C%2FP%3E%3C%2FLINGO-BODY%3E
Occasional Contributor

new to kql here, is it possible to build a  query that search's across logs looking for machines that connected to any of ip addresses in the watchlist?  Any examples ? Plan would be to turn that query into a log analytic rule to create events and eventually a playbook.

 

thanks

8 Replies

@roadruner Here is a simple example of how to do this.  I created a CSV file that has all the IPAddresses I have cleared and uploaded that into the Watchlist using "ClearedIPAddreses" as the alias.

 

let ClearedIPAddresses=_GetWatchlist('ClearedIPAddresses');
Heartbeat
| join ClearedIPAddresses on $left.ComputerIP == $right.IPAddress

@Gary Bushey  Hi, i tried that query with alias of test1 which is alias of watchlist and received an error,

 

let ClearedIPAddresses=_GetWatchlist('test1');
Heartbeat
join ClearedIPAddresses on $left.ComputerIP == $right.IPAddress
 
error is 'join' operator: failed to resolve Column named "IPAddress"
 
my csv file has the name IP Addresses in first cell then next cells below the actual ip addresses.
 
What do you mean by cleared? The ip's I would have in my list would be IOC's, thus checking to see if any machines were hitting them.
thanks again

@roadruner You CSV file would need to have the column headers in the first row.  One of mine was "IPAddresses", you would need to substitute whatever you called your columns for that.

 

I used the term "cleared" only because my watchlist contained those IP Addresses that I want to allow.  You can call you watchlist whatever makes sense to you.

@Gary Bushey Thanks I tracked the error I had, which was the columns. It runs with no errors now. I did run a quick test and hit one of the ip's  in watchlist and then ran the query and no results found. Does the query search out all of sentinel? I tried just putting CommonSecurityLog to see if it would just search through those logs, since that's where the hit should be.

Here is what i tried, didn't work.

 

CommonSecurityLog

let ClearedIPAddresses=_GetWatchlist('test1');
Heartbeat
| join ClearedIPAddresses on $left.ComputerIP == $right.IPAddresses

 

@roadruner It will only search the one table. 

 

There really is no way to search all tables for multiple values.  There is the "search" command, https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/searchoperator, that allows you to search for a single term although I do not know if this can be used in a Analytic rule or not.

@Gary Bushey Hmm ok, thanks.  How can i search one table? say CommonSecurityLog

I tried this but no dice.

 

CommonSecurityLog

let ClearedIPAddresses=_GetWatchlist('test1');
Heartbeat
| join ClearedIPAddresses on $left.ComputerIP == $right.IPAddress

best response confirmed by roadruner (Occasional Contributor)
Solution

@roadruner This is the starting query for something like that.

 

 

let ClearedIPAddresses=_GetWatchlist('test1');
CommonSecurityLog
| join ClearedIPAddresses on $left.SourceIP== $right.IPAddress

@Gary Bushey Thanks! This worked. Just replaced sourceip to destip. and .found the test hits to the list. either way works.