Minemeld Threat Intel Integration to Sentinel

[GabrielNecula]

Hello guys,

I have deployed a Minemeld server in Azure, I'm pulling free threat intel in there. Processing it, then using the Microsoft Security Graph extension to forward it to Microsoft. Turned the Threat Intel Connector on and now I have the Threat Intel in the LogAnalytics space.

There are two issues I have, in order:

1. Currently, with threat intel of type IP, I get the IP in a field called ExternalIndicatorID. A sample value for this is: IPv4:36.119.0.0-36.119.255.255 . As you can see, we have IPv4: then a range of IPs follows. The problem is this is something that's very impractical to use from an analytics point of view. I have to write the query in such a way to ignore the "IPv4:" and then also be able to interpret range. This is impractical and the preview Threat Intel rules offered by Microsoft do not use that field. They instead use NetworkIP, NetworkDestinationIP, NetworkSourceIP ....whichever of the three they find with a value. For me however, those values are empty.

Apparently this is something that must be changed with the Minemeld processor so that it does not merge IPs and generate ranges. I have not found a way to do that.

Has anyone managed to do that or otherwise any other workarounds to be able to consume Minemeld IP Threat Intel in Sentinel?

2. The second thing and I'm not completely sure here as nr 1 was a much bigger priority, is the Microsoft Security Graph extension for Minemeld only able to consume URLs, Domains and IPs? No emails, hashes, etc?

I have also asked on Palo Alto's board, however I'm really curious and could use a hand from someone who managed to already do this.

Thank you!

[#2]

@JoachimLassus 

Ah ok, sorry as I don't have the the raw data I assumed the CIDR was in multiple rows.  You will probably have to check with a IP4_is_match in that case, use this as a starting point 

let tiIP = toscalar(ThreatIntelligenceIndicator
//| where isnotempty(NetworkCidrBlock)
// add in a fake CICRblock
| summarize by  NetworkCidrBlock = "193.228.91.0/26");
SigninLogs
// add a fake IP
| project IPAddress = "193.228.91.63"
| summarize by IPAddress, tiIP, found_IP_inCIDRblock=ipv4_is_match(IPAddress, tiIP)

When I pass in .63 its within the CIDR range, so the last column == true 

IPAddress	__scalar_89df8fb8a4a94d418dfea734f7a12561	found_IP_inCIDRblock
193.228.91.63	193.228.91.0/26	true

else, when I use .64

IPAddress	__scalar_87a8d226aae14e3bb6dd5fd09ff9472e	found_IP_inCIDRblock
193.228.91.64	193.228.91.0/26	false

You may even need to use this in a Function - as an idea? 

let tiIP = toscalar(ThreatIntelligenceIndicator
| where isnotempty(NetworkCidrBlock)
| summarize by  NetworkCidrBlock);
SigninLogs
| project IPAddress
| summarize by IPAddress, tiIP, found_IP_inCIDRblock=ipv4_is_match(IPAddress, tiIP)​

or even just get the latest CIDR block, with arg_max?

let tiIP = toscalar(ThreatIntelligenceIndicator
| where isnotempty(NetworkCidrBlock)
| summarize arg_max(NetworkCidrBlock,*) );
SigninLogs
| project IPAddress
| summarize by IPAddress, tiIP, found_IP_inCIDRblock=ipv4_is_match(IPAddress, tiIP)

[#3]

@CliveWatson 

Thank you, that works for a single NetworkCIDRblock value, when you import TI from Minemeld it comes with a lot of records with different CIDR block values. 

1. 141.98.81.0/24

2. 94.102.51.0/24

...and so on.

The latest query seems to only take the first NetworkCIDRblock record into consideration so if there's an IPAddress in the Signinlogs between 94.102.51.0-94.102.51.255 it wouldn't trigger. 

[#4]

@JoachimLassus 

Are we slowly getting to an answer, sorry its taken a while but its hard to do without seeing or having this data?

I have used a DataTable to emulate 2 rows of the data I think you are seeing, using 3 just columns: TimeGenerated, NetworkCidrBlock and NetworkIP.  Now I have made an assumption that the NetworkIP appears in the same rows as a CIDR block and its an IP that I can use for the Join?????  Is it one somewhere in the CIDR block or the first one?

let ThreatIntelligenceIndicator = 
datatable (timeGenerated:datetime,NetworkCidrBlock:string, NetworkIP:string)
[
    datetime("10/7/2020, 1:25:34.971 PM"),"193.228.91.0/26","193.228.91.0",
    datetime("10/7/2020, 1:25:35.971 PM"),"94.102.51.0/36","94.102.51.0"
]
;
ThreatIntelligenceIndicator
| join 
(
SigninLogs
| project IPAddress = "94.102.51.0"
) on $left.NetworkIP == $right.IPAddress
| summarize by IPAddress, NetworkCidrBlock, found_IP_inCIDRblock=ipv4_is_match(IPAddress, NetworkCidrBlock)

You should just be able to take the above and run this part - as shown below (if my assumption that the NetworkIP == IPAddress in SigninLogs is right?), if not we need another column to join the data on:

ThreatIntelligenceIndicator
| join 
(
SigninLogs
//| project IPAddress = "94.102.51.63"
) on $left.NetworkIP == $right.IPAddress
| summarize by IPAddress, NetworkCidrBlock, found_IP_inCIDRblock=ipv4_is_match(IPAddress, NetworkCidrBlock)

It would help to see a few lines of the real data from Minemeld - perhaps you can run

ThreatIntelligenceIndicator

| limit 5

And export this to Excel in the GUI and send me a private message with it, if you don't want to share it here?

Thanks Clive

[#5]

@CliveWatson 

Tried to send you a PM but it keeps timing out. Here's the output from the query:

TenantId	TimeGenerated	SourceSystem	Action	ActivityGroupNames	AdditionalInformation	ApplicationId	AzureTenantId	ConfidenceScore	Description	ExternalIndicatorId	ExpirationDateTime	IndicatorId	ThreatType	Active	MalwareNames	Tags	TrafficLightProtocolLevel	NetworkCidrBlock	Type
56241ceb-e7c3-4e86-a1d1-5b811ca58c07	2020-10-06T20:36:44.366Z	SecurityGraph	alert	[]			6bfdb47a-cb3e-4b91-854a-9d201e501f6a	50	IPv4 indicator from ET.compromised_ips	IPv4:162.246.232.59-162.246.232.59	2020-11-04T20:35:13.158Z	AFA511155E730E2B3F1F94B17AF122047A1A06725768DECD237FF4A9DA364349	Malware	TRUE	[]	[]	green	162.246.232.59/32	ThreatIntelligenceIndicator
56241ceb-e7c3-4e86-a1d1-5b811ca58c07	2020-10-06T20:36:44.421Z	SecurityGraph	alert	[]			6bfdb47a-cb3e-4b91-854a-9d201e501f6a	100	IPv4 indicator from dshield.block	IPv4:94.102.56.0-94.102.56.237	2020-11-04T20:35:13.165Z	8F932C6DE5E12C584EB97336A31D4AE784F8A5D4FCF9B8BFAF66FB649DE451A7	Malware	TRUE	[]	[]	green	94.102.56.232/30	ThreatIntelligenceIndicator
56241ceb-e7c3-4e86-a1d1-5b811ca58c07	2020-10-06T20:36:44.46Z	SecurityGraph	alert	[]			6bfdb47a-cb3e-4b91-854a-9d201e501f6a	100	IPv4 indicator from dshield.block	IPv4:94.102.56.239-94.102.56.255	2020-11-04T20:35:13.166Z	319E70E85C9E8EA6FF09144DAAAF9834153EEDF4FAB80D563C3AFD4B474D54C6	Malware	TRUE	[]	[]	green	94.102.56.239/32	ThreatIntelligenceIndicator
56241ceb-e7c3-4e86-a1d1-5b811ca58c07	2020-10-06T20:36:44.48Z	SecurityGraph	alert	[]			6bfdb47a-cb3e-4b91-854a-9d201e501f6a	100	IPv4 indicator from dshield.block	IPv4:45.129.33.0-45.129.33.255	2020-11-04T20:35:13.144Z	98608B1CF8142DB574A10AAFFB7FABF4941A5E10D389F884ED59C951A6B374CE	Malware	TRUE	[]	[]	green	45.129.33.0/24	ThreatIntelligenceIndicator
56241ceb-e7c3-4e86-a1d1-5b811ca58c07	2020-10-06T20:36:44.492Z	SecurityGraph	alert	[]			6bfdb47a-cb3e-4b91-854a-9d201e501f6a	50	IPv4 indicator from ET.compromised_ips	IPv4:93.123.16.135-93.123.16.135	2020-11-04T20:35:13.158Z	0C29DB110C776891A25CCFF4306B1765983CD84F8F5D871EEE64B46E0E239857	Malware	TRUE	[]	[]	green	93.123.16.135/32	ThreatIntelligenceIndicator

[#6]

@JoachimLassus You will need to add the parse_ipv4_mask and parse_ipv4 commands before doing the compare.  Like @CliveWatson I do not have the needed data to completely test but from my simple tests, this appears to work

ThreatIntelligenceIndicator
| where isnotempty(NetworkCidrBlock)
| project NetworkCidrBlock = "193.228.91.0/26"
| project justIP = tostring(split(NetworkCidrBlock,"/").[0]) , prefixIP = toint(split(NetworkCidrBlock,"/").[1])
| extend compareIP = parse_ipv4_mask(justIP,prefixIP)
| join (
SigninLogs
| project IPAddress = "193.228.91.0"
//use parse_ipv4 to convert IP to long
| extend compareIP = parse_ipv4(IPAddress)
)
on $left.compareIP== $right.compareIP

[#7]

@Gary Bushey 

Gary, Minemeld gives lots of different NetworkCIDRBlocks records that are considered malicious. Some contains 1 host but many contains several hosts. For example 2 records could looks like this:

1. 94.102.56.232/30 which gives you the following range 94.102.56.232 to 94.102.56.235

2. 94.102.56.239/32 which gives you 94.102.56.239

So the query would need to check each IPAddress from Signinglogs and see if they exist within those ranges. 

[#8]

@JoachimLassus Unless I am missing something the query I posted in my last response should do just that.

[#9]

@Gary Bushey 

If I use one of the previous examples and create a datatable like this:

let ThreatIntelligenceIndicator = 
datatable (timeGenerated:datetime,NetworkCidrBlock:string, NetworkIP:string)
[
 datetime("10/7/2020, 1:25:34.971 PM"),"193.228.91.0/26","193.228.91.0"
]
;
ThreatIntelligenceIndicator
| extend CompareIP2 = parse_ipv4(NetworkIP)
| project justIP = tostring(split(NetworkCidrBlock,"/").[0]) , prefixIP = toint(split(NetworkCidrBlock,"/").[1]), CompareIP2
| extend compareIP = parse_ipv4_mask(justIP,prefixIP)

The output for parse_ipv4(NetworkIP) is 3,252,968,192 and the output for parse_ipv4_mask(justIP,prefixIP) is also 3,252,968,192 .

If I change NetworkIP in the table to be 193.228.91.1 then the output for parse_ipv4(NetworkIP) is 3,252,968,193.

That would mean if we have 193.228.91.1 in the Signinlogs then the join from your query wouldn't work, right? 

[#10]

@JoachimLassus You are correct, I told you to use the wrong command.  ipv4_compare is what you need to use (BTW, also found out that you can use CIDR notation in parse_ipv4 so parse_ipv4_mask is not needed.

https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/ipv4-comparefunction

Now we just need to figure out how to use the ip4_compare in a join.  The best I could come up with, since joins only allows equality comparisons, is something like this.  Hopefully someone better with KQL can come up with something a bit more elegent.

let X = datatable(Key:string, Value1:string)
[
    'a',"193.228.91.1",
];
let Y = datatable(Key:string, Value2:string)
[
    'a',"193.228.91.0/26",
];
let Z =X | join Y on Key;
Z | where ipv4_compare( Value1, Value2)==0

[#11]

I think the following query gets the correct result, so far the testing hasnt shown anything incorrect anyway.

SigninLogs

| extend sign_get_three_octet=extract("([0-9]*.[0-9]*.[0-9]*)", 1, IPAddress)

//| project sign_get_three_octet , IPAddress

| join kind=inner (

ThreatIntelligenceIndicator

| extend threat_get_three_octet=extract("([0-9]*.[0-9]*.[0-9]*)", 1, NetworkCIDRBlock)

//| project threat_get_three_octet , NetworkCIDRBlock

) on $left.sign_get_three_octet== $right.threat_get_three_octet

| where ipv4_is_match(IPAddress,NetworkCIDRBlock)

| project IPAddress, NetworkCIDRBlock

A million thanks to @CliveWatson and @Gary Bushey 

[#12]

Very pleased you got it working, and pleased the work Gary did and my tip to match on octets helped ;)

[#13]

@GabrielNeculaHere is another trick

//datatable or watchlist can be added here, in this example i use static datatable

let IPLookup = datatable(cidr:string, cidr_name:string)
[
"16.168.0.0/16", "cidr_name_1",
"16.167.0.0/16", "cidr_name_2",
];
TABLEwithIP
| evaluate ipv4_lookup(IPLookup, from_address_s, cidr, return_unmatched = false)