Minemeld Threat Intel Integration to Sentinel

I think the following query gets the correct result, so far the testing hasnt shown anything incorrect anyway.

JoachimLassus You are correct, I told you to use the wrong command.  ipv4_compare is what you need to use (BTW, also found out that you can use CIDR notation in parse_ipv4 so parse_ipv4_mask is not needed.

https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/ipv4-comparefunction

Now we just need to figure out how to use the ip4_compare in a join.  The best I could come up with, since joins only allows equality comparisons, is something like this.  Hopefully someone better with KQL can come up with something a bit more elegent.

let X = datatable(Key:string, Value1:string)
[
    'a',"193.228.91.1",
];
let Y = datatable(Key:string, Value2:string)
[
    'a',"193.228.91.0/26",
];
let Z =X | join Y on Key;
Z | where ipv4_compare( Value1, Value2)==0

GaryBushey 

let ThreatIntelligenceIndicator = 
datatable (timeGenerated:datetime,NetworkCidrBlock:string, NetworkIP:string)
[
 datetime("10/7/2020, 1:25:34.971 PM"),"193.228.91.0/26","193.228.91.0"
]
;
ThreatIntelligenceIndicator
| extend CompareIP2 = parse_ipv4(NetworkIP)
| project justIP = tostring(split(NetworkCidrBlock,"/").[0]) , prefixIP = toint(split(NetworkCidrBlock,"/").[1]), CompareIP2
| extend compareIP = parse_ipv4_mask(justIP,prefixIP)

JoachimLassus Unless I am missing something the query I posted in my last response should do just that.

GaryBushey 

Gary, Minemeld gives lots of different NetworkCIDRBlocks records that are considered malicious. Some contains 1 host but many contains several hosts. For example 2 records could looks like this:

1. 94.102.56.232/30 which gives you the following range 94.102.56.232 to 94.102.56.235

2. 94.102.56.239/32 which gives you 94.102.56.239

So the query would need to check each IPAddress from Signinglogs and see if they exist within those ranges. 

JoachimLassus You will need to add the parse_ipv4_mask and parse_ipv4 commands before doing the compare.  Like CliveWatson I do not have the needed data to completely test but from my simple tests, this appears to work

ThreatIntelligenceIndicator
| where isnotempty(NetworkCidrBlock)
| project NetworkCidrBlock = "193.228.91.0/26"
| project justIP = tostring(split(NetworkCidrBlock,"/").[0]) , prefixIP = toint(split(NetworkCidrBlock,"/").[1])
| extend compareIP = parse_ipv4_mask(justIP,prefixIP)
| join (
SigninLogs
| project IPAddress = "193.228.91.0"
//use parse_ipv4 to convert IP to long
| extend compareIP = parse_ipv4(IPAddress)
)
on $left.compareIP== $right.compareIP

CliveWatson 

Tried to send you a PM but it keeps timing out. Here's the output from the query:

TenantId	TimeGenerated	SourceSystem	Action	ActivityGroupNames	AdditionalInformation	ApplicationId	AzureTenantId	ConfidenceScore	Description	ExternalIndicatorId	ExpirationDateTime	IndicatorId	ThreatType	Active	MalwareNames	Tags	TrafficLightProtocolLevel	NetworkCidrBlock	Type
56241ceb-e7c3-4e86-a1d1-5b811ca58c07	2020-10-06T20:36:44.366Z	SecurityGraph	alert	[]			6bfdb47a-cb3e-4b91-854a-9d201e501f6a	50	IPv4 indicator from ET.compromised_ips	IPv4:162.246.232.59-162.246.232.59	2020-11-04T20:35:13.158Z	AFA511155E730E2B3F1F94B17AF122047A1A06725768DECD237FF4A9DA364349	Malware	TRUE	[]	[]	green	162.246.232.59/32	ThreatIntelligenceIndicator
56241ceb-e7c3-4e86-a1d1-5b811ca58c07	2020-10-06T20:36:44.421Z	SecurityGraph	alert	[]			6bfdb47a-cb3e-4b91-854a-9d201e501f6a	100	IPv4 indicator from dshield.block	IPv4:94.102.56.0-94.102.56.237	2020-11-04T20:35:13.165Z	8F932C6DE5E12C584EB97336A31D4AE784F8A5D4FCF9B8BFAF66FB649DE451A7	Malware	TRUE	[]	[]	green	94.102.56.232/30	ThreatIntelligenceIndicator
56241ceb-e7c3-4e86-a1d1-5b811ca58c07	2020-10-06T20:36:44.46Z	SecurityGraph	alert	[]			6bfdb47a-cb3e-4b91-854a-9d201e501f6a	100	IPv4 indicator from dshield.block	IPv4:94.102.56.239-94.102.56.255	2020-11-04T20:35:13.166Z	319E70E85C9E8EA6FF09144DAAAF9834153EEDF4FAB80D563C3AFD4B474D54C6	Malware	TRUE	[]	[]	green	94.102.56.239/32	ThreatIntelligenceIndicator
56241ceb-e7c3-4e86-a1d1-5b811ca58c07	2020-10-06T20:36:44.48Z	SecurityGraph	alert	[]			6bfdb47a-cb3e-4b91-854a-9d201e501f6a	100	IPv4 indicator from dshield.block	IPv4:45.129.33.0-45.129.33.255	2020-11-04T20:35:13.144Z	98608B1CF8142DB574A10AAFFB7FABF4941A5E10D389F884ED59C951A6B374CE	Malware	TRUE	[]	[]	green	45.129.33.0/24	ThreatIntelligenceIndicator
56241ceb-e7c3-4e86-a1d1-5b811ca58c07	2020-10-06T20:36:44.492Z	SecurityGraph	alert	[]			6bfdb47a-cb3e-4b91-854a-9d201e501f6a	50	IPv4 indicator from ET.compromised_ips	IPv4:93.123.16.135-93.123.16.135	2020-11-04T20:35:13.158Z	0C29DB110C776891A25CCFF4306B1765983CD84F8F5D871EEE64B46E0E239857	Malware	TRUE	[]	[]	green	93.123.16.135/32	ThreatIntelligenceIndicator

JoachimLassus 

Are we slowly getting to an answer, sorry its taken a while but its hard to do without seeing or having this data?

I have used a DataTable to emulate 2 rows of the data I think you are seeing, using 3 just columns: TimeGenerated, NetworkCidrBlock and NetworkIP.  Now I have made an assumption that the NetworkIP appears in the same rows as a CIDR block and its an IP that I can use for the Join?????  Is it one somewhere in the CIDR block or the first one?

let ThreatIntelligenceIndicator = 
datatable (timeGenerated:datetime,NetworkCidrBlock:string, NetworkIP:string)
[
    datetime("10/7/2020, 1:25:34.971 PM"),"193.228.91.0/26","193.228.91.0",
    datetime("10/7/2020, 1:25:35.971 PM"),"94.102.51.0/36","94.102.51.0"
]
;
ThreatIntelligenceIndicator
| join 
(
SigninLogs
| project IPAddress = "94.102.51.0"
) on $left.NetworkIP == $right.IPAddress
| summarize by IPAddress, NetworkCidrBlock, found_IP_inCIDRblock=ipv4_is_match(IPAddress, NetworkCidrBlock)

You should just be able to take the above and run this part - as shown below (if my assumption that the NetworkIP == IPAddress in SigninLogs is right?), if not we need another column to join the data on:

ThreatIntelligenceIndicator
| join 
(
SigninLogs
//| project IPAddress = "94.102.51.63"
) on $left.NetworkIP == $right.IPAddress
| summarize by IPAddress, NetworkCidrBlock, found_IP_inCIDRblock=ipv4_is_match(IPAddress, NetworkCidrBlock)

It would help to see a few lines of the real data from Minemeld - perhaps you can run

CliveWatson 

Thank you, that works for a single NetworkCIDRblock value, when you import TI from Minemeld it comes with a lot of records with different CIDR block values. 

1. 141.98.81.0/24

2. 94.102.51.0/24

...and so on.

The latest query seems to only take the first NetworkCIDRblock record into consideration so if there's an IPAddress in the Signinlogs between 94.102.51.0-94.102.51.255 it wouldn't trigger. 

JoachimLassus 

Ah ok, sorry as I don't have the the raw data I assumed the CIDR was in multiple rows.  You will probably have to check with a IP4_is_match in that case, use this as a starting point 

let tiIP = toscalar(ThreatIntelligenceIndicator
//| where isnotempty(NetworkCidrBlock)
// add in a fake CICRblock
| summarize by  NetworkCidrBlock = "193.228.91.0/26");
SigninLogs
// add a fake IP
| project IPAddress = "193.228.91.63"
| summarize by IPAddress, tiIP, found_IP_inCIDRblock=ipv4_is_match(IPAddress, tiIP)

When I pass in .63 its within the CIDR range, so the last column == true 

let tiIP = toscalar(ThreatIntelligenceIndicator
| where isnotempty(NetworkCidrBlock)
| summarize by  NetworkCidrBlock);
SigninLogs
| project IPAddress
| summarize by IPAddress, tiIP, found_IP_inCIDRblock=ipv4_is_match(IPAddress, tiIP)​

let tiIP = toscalar(ThreatIntelligenceIndicator
| where isnotempty(NetworkCidrBlock)
| summarize arg_max(NetworkCidrBlock,*) );
SigninLogs
| project IPAddress
| summarize by IPAddress, tiIP, found_IP_inCIDRblock=ipv4_is_match(IPAddress, tiIP)

CliveWatson 

Thanks for you response, correct me if I'm wrong but I believe that query only considers if 193.228.91.0 exists in the Signinlogs.

193.228.91.0/24 would give you a range of 193.228.91.0 - 193.228.91.63 so if someone with IP 193.228.91.5 signs in it wouldn't trigger an alert.

JoachimLassus 

Ah ok, my entries in that column are empty, so it could work like this?  Checking each TI entry against the Signinlogs  

ThreatIntelligenceIndicator
//| where isnotempty(NetworkCidrBlock)
| project NetworkCidrBlock = "193.228.91.0/26"
| project justIP = tostring(split(NetworkCidrBlock,"/").[0]) , prefixIP = toint(split(NetworkCidrBlock,"/").[1])
| join (
SigninLogs
| project IPAddress = "193.228.91.0"
)
on $left.justIP == $right.IPAddress
| project parse_ipv4_mask(justIP, prefixIP)

 You'll need to uncomment line 2 and remove lines 3 & 7 

ThreatIntelligenceIndicator
| where isnotempty(NetworkCidrBlock)
//| project NetworkCidrBlock = "193.228.91.0/26"
| project justIP = tostring(split(NetworkCidrBlock,"/").[0]) , prefixIP = toint(split(NetworkCidrBlock,"/").[1])
| join (
SigninLogs
//| project IPAddress = "193.228.91.0"
)
on $left.justIP == $right.IPAddress
| project parse_ipv4_mask(justIP, prefixIP)

CliveWatson 

The TI data from Minemeld have these columns

So we get range of bad IP's, wouldn't you have to cycle through each IP to see if it exists in / to join the Signinlogs table?

ThreatIntelligenceIndicator
| extend TI_ipEntity = iff(isnotempty(NetworkIP), NetworkIP, NetworkDestinationIP)
| where isnotempty( TI_ipEntity)
//| project NetworkIP = "2.2.2.2/31"
| extend TI_ipEntity = iif(NetworkIP has "/", tostring(split(NetworkIP,"/").[0]),NetworkIP)
| project TI_ipEntity, NetworkIP
| join (
SigninLogs
//| project IPAddress = "2.2.2.2"
)
on $left.TI_ipEntity == $right.IPAddress
| project IPAddress, NetworkIP
| extend NetworkIP = iif(NetworkIP has "/", tostring(split(NetworkIP,"/").[1]),NetworkIP)
| project parse_ipv4_mask(IPAddress, toint(NetworkIP))

You can probably check for a IP with a prefix mask "/" and filter those, something like the above?

JoachimLassus 

Sorry which bit isn't working?  https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/parse-ipv4-maskfunction  this will work from any Workspace that Azure Sentinel is associated with.  However you will need a record in the Workspace that has a IP and prefix mask to supply as a parameter:

SigninLogs
| summarize by IPAddress, parse_ipv4_mask(IPAddress,31)

GaryBushey  Thanks for your reply. I'm still struggling with this, parse_ipv4_mask doesn't seem to be available within Azure Sentinel or am I missing something?

CliveWatson  Is this something you have looked at before?