May 16 2019 01:11 PM
Pull file hashes (SHA1) from Malware Information Sharing Platform (MISP) and push them to Microsoft Defender ATP
5 Minutes
Low complexity
Enterprises use threat intelligence to enrich their cyber security telemetry as well as to detect and block attacks. Microsoft Defender ATP supports blocking capabilities through the portal using the indicators page and the indicators API. In a previous blog, we explained how to generally use the indicators API.
In our commitment to listen to customer feedback, we’re going to demonstrate how to leverage the power of Malware Information Sharing Platform (MISP) to further augment the protection offering of Microsoft Defender ATP.
MISP is a free and open source threat sharing platform. It is designed to help share threat intelligence information such as cyber security indicators, vulnerability information, and others.
In this blog, we will demonstrate an easy way to automatically pull the indicators from the MISP platform and push them into the custom IoC list of Microsoft Defender ATP.
We’ll guide you on how to pull SHA1 hashes from the past four days from MISP, and push them as indicators for Microsoft Defender ATP to detect. We’ll push the indicators with an expiration date set to seven days from the time the script is run to ensure the freshness of indicators.
NOTE: This step is identical to the process detailed in our previous blog. If you’ve already done it, skip this step.
If you haven’t created an app:
If you’ve already created an app that you’re going to reuse for this demonstration:
Add TI write permissions
Done! You have successfully added the required permissions to the application.
Step 2: Get your MISP URL and Authorization key
To use MISP API, you will your MISP URL and authorization key. You can get the URL from MISP web interface and the API key from the Automation section of the MISP web interface. See below:
Download the zip file at the bottom of this blog and extract the “Get-MISP-Hash.ps1” script.
Make sure the script in the same directory (folder) as your “Get-Token.ps1” script.
Now lets run the script!
Let’s say your MISP URL is 10.0.0.1 and the key is Yn81mP1ck13Ric4.
Open PowerShell and run the following command:
.\Get-Misp-Hash.ps1 -title “Sha1 from MISP” -mispUrl “10.0.0.1” -authKey “Yn81mP1ck13Ric4”
And that’s it, you are done! You can see the new hashes on your Microsoft Defender Security Portal by clicking on Settings, then look for “Indicators” on the menu on the left of the screen.
The script also offers additional (optional) fields you may want to use to control the indicators and how the MDATP handles them:
Parameter |
Description |
Default value |
-title |
The alert’s title |
- |
-mispUrl |
Your MISP URL from the MISP platform |
- |
-authKey |
Your MISP authorization key |
- |
-action (optional) |
Choose one: Alert/AlertAndBlock/Allowed |
Alert |
-severity (optional) |
Choose one: Informational/Low/Medium/High |
Informational |
-description (optional) |
the alert’s description |
- |
-recommendedAction (optional) |
the recommended action to remediate the alert |
- |
-expiration (optional) |
number of days the hashes will expire and deleted from the MDATP |
7 |
Another example that utilizes all of the options is shown below all in the same line:
.\Get-MISP-Hash.ps1 -title "Sha1 from MISP" -mispUrl “10.0.0.1 ” -authKey “Yn81mP1ck13Ric4” -action AlertAndBlock -severity High -description "Sha1 from the MISP platform loaded by automated script" -recommendedActions "Add your own recommended actions" -expiration 14
In this blog, we demonstrated how you can pull SHA1 hashes from MISP to augment the Microsoft Defender ATP protection capabilities
FAQs:
Which file hashes are supported?
This integration supports SHA1. MISP’s API only supports SHA1 and MD5 (which is relatively weak), while Microsoft Defender ATP supports SHA1 and SHA256.
Also note that “Only published events and attributes marked as IDS Signature are exported” (MISP API documentation).
How do I modify the MISP filter?
The MISP API we used to get the hashes has some filtering options.
We set the script to get hashes from the past 4 days:
https://<your_MISP_URL>/events/hids/sha1/download/false/false/false/4d
You can change it to any other number of days or hours, also you can filter by start and end date or by MISP tags you’ve set up.
For the full details, see MISP’s API DOC here: https://www.circl.lu/doc/misp/automation/ and look for the “GET /events/hids Hash” section.
Important note:
If the hash you are submitting already exists in Microsoft Defender ATP with a different expiration date or severity etc., it’s OK – it will be updated with the new value. However, if it already exists with a different “action” field value (Alert, AlertAndBlock or Allow), it will not be updated (or submitted), and the script will print the hash with the corresponding message – this is by design.
Please share your thoughts and feedback here.
Thanks,
@Itai Zur, program manager, Windows Defender ATP
@Thorsten Henking, senior program manager
@Dan Michelson, program manager, Windows Defender ATP
@Haim Goldshtein, security software engineer, Windows Defender ATP
Jun 06 2019 04:43 AM
@Haim Goldshtein - Thank you for the guide, just some clarifications from MISP's side: MISP supports a long list of hashing algorithms, I think you may have ended up on an old API documentation, the /hids endpoints are deprecated. Instead we have transitioned to the restSearch APIs (/events/restSearch and /attributes/restSearch) that allow you to search and export any of the supported types (which for hashes includes sha256, various fuzzy hashing algorithms like ssdeep, import hashes, etc).
An example of using the modern APIs:
https://<your_MISP_URL>/events/hids/sha1/download/false/false/false/4d
would be
POST to https://<your_MISP_URL>/attributes/restSearch
payload:
{
"returnFormat": "text",
"publish_timestamp": "4d",
"type": "sha1"
}
Type of course could be sha256 or ssdeep for example.
If you prefer URL parameters and GET requests it would look like this:
GET to https://<your_MISP_URL>/attributes/restSearch/returnFormat:text/type:sha1/publish_timestamp:4d
The best resource for the API is the built in documentation under event actions -> automation or alternatively the REST client with the query builder (Event actions -> REST client).
Hope this helps and thanks for the integration and associated guide!
Aug 09 2019 05:31 AM
@Haim Goldshtein
Thanks for the example code.
We tweaked it a little, as we found the indicator upload API is rate-limited, and with more than about 150 indicators returned from MISP the script would run into this and start to receive 429 error responses. In that case, ours will wait 15s and retry the indicator a few times before giving up.
There's also a new batch upload mechanism and we added a parameter (and code) to use that instead - with "-batchUpload 1". This can feed 500 at a time and is much faster.
We also added the "Get-Token.ps1" code to the main script to remove the dependency (oh, and changed to the MISP REST GET format URL per the previous Reply.)
I've attached the result with any company specifics stripped out in case it's useful. Note that some parameters are renamed: atpAction, atpTitle, atpSeverity, atpDescription, and atpRecommendedActions .
Jun 03 2020 05:29 PM - edited Jun 03 2020 05:53 PM
I keep getting an error on this script;
"Invoke-WebRequest : A parameter cannot be found that matches parameter name 'k'."
On the "$response = curl -k --header" line. I've tried playing around with the curl/Invoke-WebRequest syntax a bit but no dice.
Edit: Using Powershell 7 resolved this.
Aug 01 2022 11:20 AM
May 15 2024 11:13 AM
May 29 2024 09:40 AM
Hello,
I get this error msj from the PS Script.
Any idea?
Invoke-WebRequest : A positional parameter cannot be found that accepts argument 'Authorization:
PS Script Line
$response = curl --header $authorization --header "Accept: application/json" --header "Content-Type: application/json" hxxps://$mispUrl/events/hids/sha1/download/false/false/false/4d > JSON_DATE.txt