Microsoft Secure Tech Accelerator
Apr 03 2024, 07:00 AM - 11:00 AM (PDT)
Microsoft Tech Community
Integrating open source threat feeds with MISP and Sentinel
Published May 14 2020 10:55 AM 48K Views
Microsoft

Recently, Microsoft released an open source set of malicious file hash indicators identified as using COVID-19 themed malicious email attachments in attempted attacks against our customers. Office365 successfully blocked these attempts, but the indicators can be consumed and used by customers to further protect themselves. The feed of indicators is provided as data file on GitHub which can be consumed using MISP.

 

In this blog post I will show Azure Sentinel customers how to set up a MISP server that can receive any public feeds, including these COVID-19 indicators, and import the data into your Azure Sentinel environment. It is also possible to use this code to import MISP data into Microsoft Defender ATP as well. Haim Goldshtein has already written a blog post on doing this. Instructions here have been tested on Ubuntu 18.04 but should be applicable to many other distributions – even WSL. misp.PNG

 

The COVID-specific threat intelligence feed represents a start at sharing some of Microsoft’s COVID-related IOCs.  We will continue to explore ways to improve the data over the duration of the crisis. While some threats and actors are still best defended more discreetly, we are committed to greater transparency and taking community feedback on what types of information is most useful to defenders in protecting against COVID-related threats. This is a time limited feed. We are maintaining this feed through the peak of the outbreak to help organizations focus on recovery.

 

If you have questions or feedback on this COVID-19 feed, please email msft-covid19-ti@microsoft.com.

 

To integrate this feed with your MISP server you will need to use the following URL:

https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Sample%20Data/Feeds/Microsoft.Covid19....

 

Install Docker

The Docker project has already published comprehensive documentation on setting up the most recent version of Docker for your distribution of choice. For this blog I used the Ubuntu instructions.

The Docker MISP instance also requires ‘docker-compose’ so once you have followed the Docker install guide enter the following command.

 

sudo apt-get install docker-compose

 

 

Set up MISP Docker instance

The MISP project has published a Docker compose configuration, you can use this by first entering these commands.

 

git clone https://github.com/MISP/misp-docker
cd misp-docker

 

Next, you will need to edit the configuration file, making sure to set a strong password. If you do not set a strong enough password, you might not be able to sign into your MISP instance. This can be fixed later.

 

cp template.env .env
nano .env

 

Now the Docker image needs to be built. Run these two commands to build the image and start the container.

 

sudo docker-compose build
sudo docker-compose up

 

At this point a MISP instance will be running on port 80. You should be able to sign in and begin adding new feeds. If you are hosting this server on the Internet, you will want to look at how to secure this installation further with TLS and restrictions on access to the web front end.

 

If you are unable to login to the front end, then perhaps the password was not strong enough. You can reset the password with the following commands.

 

sudo docker exec -i -t misp_web /bin/bash
/var/www/MISP/app/Console/cake Password admin@admin.test NEWPASSWORD
exit

 

 

Add the COVID-19 feed

The next step is to add the Microsoft feed to the MISP server. There is good documentation for this but in brief click ‘Sync Actions’ on the main menu then ‘List feeds’ and click ‘Add Feed’. The address of Microsoft’s COVID-19 feed can be found above. Enter this in the URL textbox. Next you will need to select ‘Simple CSV Parsed Feed’ from the list box. Most of the text boxes can be left blank but you must set the ‘Value field(s) in the CSV’ to 2. Set the other properties to reasonable values and click Add. Make sure you have ticked the ‘Enable’ checkbox.

 

There are several other 3rd party feeds you may also want to enable and have available in your Sentinel workspace. Each of these will need to be enabled separately.

x1.png

The next step is to ensure that the feed is automatically updated. In the ‘Scheduled Tasks’ section of the Administration menu set the fetch_feeds task frequency to 1h. If you want to fetch on a quicker schedule this can be performed via a cron job.

 

You should see a new COVID-19 event appear from the Microsoft COVID-19 feed when the sync process starts.

 

Retrieve your MISP auth key

Within the MISP web interface click ‘Event Actions’ on the menu bar then select ‘Automation’. Your MISP auth key will be listed on the screen, note this down for entry into the script later.

 

Connect your MISP instance to Sentinel

Much of this section is an abridged version of the Sentinel threat intelligence feed connector and MISP to Microsoft Graph script documentation. You should review this documentation first.

 

Create an App Registration with the required permissions

In order to connect your MISP server to Sentinel you need to create an App Registration with the required permissions. This is a straightforward process but does require a user with 'Global Administrator', 'Security Administrator' or 'Security Reader' permission to grant access. In brief:

  1. Open the Application Registration Portal and click New registration on the menu bar.
  2. Enter a name, and choose Register, other options can be left with their defaults.
  3. Note down the Application (client) ID and Directory (tenant) ID. You will need to enter these into the script’s configuration file.
  4. Under Certificates & secrets, click New client secret enter a description and click Add. A new secret will be displayed. Copy this for later entry into the script.
  5. Under API permissions, choose Add a permission > Microsoft Graph.
  6. Under Application Permissions, add ThreatIndicators.ReadWrite.OwnedBy.

blogc.png

Enable the Sentinel Connector

Open your Azure Sentinel workspace, click ‘Data connectors’ and then look for the ‘Threat Intelligence Platforms’ connection. Open the connector and click Connect.

blogd.png

 

Setup the script

The script can be run on any machine that has access to your MISP infrastructure and the Microsoft Graph API. In order to reduce complexity, I ran the script on the same machine as the MISP instance.

Enter the following commands. These will create an environment for the script to run, download it from GitHub, install the necessary prerequisites and open the configuration file.

 

sudo apt-get install python3-venv
python3 -m venv mispToSentinel
cd mispToSentinel
source bin/activate
git clone https://github.com/microsoftgraph/security-api-solutions
cd security-api-solutions/Samples/MISP/
pip install -r requirements.txt
nano config.py

 

There are a few options that need to be changed in the configuration file:

  • Under the graph_auth key enter the details from the AAD App Registration earlier.
  • Set the ‘<targetProduct>’ to be ‘Azure Sentinel’.
  • I added a # comment at the start of each line in the misp_event_filters section to effectively disable any filtering, all data from the MISP server will be available in Sentinel.
  • Set ‘<action>’ to ‘alert’.
  • Enter you MISP auth key in ‘<misp key>’ and URL in ‘<misp url>’.
  • Finally set the lifetime for this data, I would recommend 30-60 days depending on your use case.

You can now run the script to pull data from the MISP instance and push into your Sentinel workspace.

 

python script.py

 

After a few minutes you should be able to query the ThreatIntelligenceIndicator table in your Sentinel workspace.

bloge.png

Use the data

Now the data is in your Sentinel workspace you can easily search for matching hashes in a variety of datasets. As an example, this query will examine the SecurityEvent table for matching hashes.

 

let BadHashes=ThreatIntelligenceIndicator
| summarize by FileHashValue;
SecurityEvent
| where FileHash in (BadHashes)
| count

 

26 Comments
Copper Contributor

Thanks for the detailed explanation, that really helps.

Is there also a way to use detected malware e-mails from the Office365 quarantine queue for feeding threat IoC's into MISP ?

 

TIA,

Ewald...

Microsoft

To get the data into your MISP server you would likely need to write some custom code. Here are a few resources that might help you.

 

https://docs.microsoft.com/en-us/powershell/module/exchange/get-quarantinemessage?view=exchange-ps

 

Microsoft

I got some build errors:

ModuleNotFoundError: No module named 'importlib.util' 

sudo docker-compose build

Solution:

fail to build main branch of misp-modules and just edit misp-docker/web/Dockerfile as below  to fix it

 

add 

 

git checkout tags/v2.4.121 && \

below :

 
# Install MISP Modules
WORKDIR /opt
RUN git clone https://github.com/MISP/misp-modules.git
RUN cd misp-modules && \
git checkout tags/v2.4.121 && \

 

 

 

Copper Contributor

I am getting errors trying to run the install script on Ubuntu 20. Does this only work on 19 or 18? The dependencies that are missing, seem not to be available in version 20. Am I missing something or is there a workaround?

 

Zachery88p_0-1647518170495.png

 

Microsoft

Looks like its failing to compile a package called cffi. I would report this on the MISP docker issues page https://github.com/MISP/misp-docker

Copper Contributor

I used an older version of Ubuntu and that worked, but now I'm running into a syntax error on the script.

 

XXXXXXXXX:~/mispToSentinel/security-api-solutions/Samples/MISP$ python script.py
File "script.py", line 28
**{field: event[field] for field in REQUIRED_GRAPH_METADATA},
^
SyntaxError: invalid syntax

Copper Contributor

Really good article! Congratulations.

Does anyone know if there is possible to receive all those MISP IoCs in Sentinel, tagged automatically with a MISSP, for example?

Copper Contributor

I am having a strange issue. I have set this up and ran the first run work perfectly. after that it seems to get stuck 

The Output is 

"fetching & parsing data from misp...
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.17.0.7'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
warnings.warn(
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.17.0.7'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
warnings.warn(
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.17.0.7'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
warnings.warn(
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.17.0.7'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
warnings.warn( "

then is just stops and does not move. 

I have waited over an hour and no movement.

the CPU is not working about 35% the network is low but the disk seems to be working hard. 

the MISP server and the solution server are the same host. is this the issue?

please advise

Copper Contributor

Thank you very much for the article.

How do we specify to which workspace the logs should be sent? What if we have more than one Sentinel instance?

 

Thanks

Brass Contributor

I also got this set up and working first time. Now I'm getting an error when trying to run the script. Same as someone else mentioned above:

 

XXXXXXXXX:~/mispToSentinel/security-api-solutions/Samples/MISP$ python script.py
File "script.py", line 28
**{field: event[field] for field in REQUIRED_GRAPH_METADATA},
^
SyntaxError: invalid syntax

 

When it was working before the XXXXXXXXX:~/mispToSentinel/security-api-solutions/Samples/MISP$ was precluded by (mispToSentinel). Not sure how to get back to that shell?

 

Do I need to log in to the docker image first? Confused! Much thanks.

Copper Contributor

I had the same issues as you all above, I had to revert to Ubuntu 18.04, which solved some issues, and that you must use Python 3.6.9 or lower, check with python3 -V, if it is too high, remove python3, and then reinstall python3.6.9 using 'sudo apt-get install python3.6', all of these dependencies are broken and outdated,

 

My issue is, MISP script is saying current batch took xxxx time, fine, then it says completed, xxxx indicators pushed to Sentinel, and yet I check my TI workspace, and there is 0 new indicators there, has the MS Graph dependency expired now too?

 

I can't believe how many features the MS suite has but yet fails to integrate the basics like TI platforms.

Brass Contributor

@robeving Could you give us some help, please? haha. I tried reverting back to python3.6, but still got the same error. I'm running this on an Ubuntu 20.04 box. It worked the first time (yesterday). And I was even able to run it a few times while MISP continued downloading TIs, but the shell was showing (mispToSentinel) before the regular misp@(machinename). It doesn't work when the shell is not showing (mispToSentinel) prior to running the command and I have no idea how to get back to that. 

Brass Contributor

Ah. I finally figured it out. And it is working with Python 3.8. To run the script successfully after you've logged out of the terminal and are coming back, you first need to:

 

cd mispToSentinel
source bin/activate

 then 

cd security-api-solutions/Samples/MISP/

then 

python script.py

If you don't "source bin/activate" first prior to attempting to run the script, you will continue to get the syntax error. I'm certainly no linux expert, but this did it for me. You'll know you did it correctly because you'll see (mispToSentinel) prior to your username@machinename in the shell.

 

Now if there were only some way to automate this process to run every x hours/days, that'd be really useful. Time to play around with my Azure Automation account. 

 

Copper Contributor

So yeah that’s simple enough just use &&

 

cd mispToSentinel && source bin/activate && cd security-api-solutions/Samples/MISP/ && python3 script.py

 

i used python3 because that doesn’t throw errors for me
Brass Contributor

New day and a new issue. Today when I'm running the script, it sits there and eventually says "Killed." I did add all of the default feeds to MISP and enabled them. But that's the only change I've made besides regular updates to the machine. Is it possible it's too much for the script to handle? I bumped the machine size up in Azure and it didn't help. I've tried running both python and python3.

Brass Contributor

May have just found out the issue. I'm seeing this in the MISP logs: 

 

Event fetch potential memory exhaustion.
During the fetching of events, a large event (#1547) was detected that exceeds the available PHP memory.
Consider raising the PHP max_memory setting to at least 10265M 

Brass Contributor

I changed the memory value in PHP.ini file but I am still having this issue. Any guidance would be appreciated. Perhaps I need to filter out certain events if they are too large or cut back on the number of subscribed feeds. Disable and enable as needed to keep it in parameters. This may be more of a question for Misp github.

Brass Contributor

Figured it out again. Was just attempting to send way too much data at once after enabling all those feeds. What I found helpful was using event filter in the script to tell MISP that I only wanted events between certain dates, so I could import smaller chunks of data at a time. Those filters are date_from and date_to. It also helped in my case after the machine had been fully updated to use the command: 'PYTHONHASHSEED=0 python3 script.py,' which is described in the documention here.

Copper Contributor

Setup fails after i run >pip install -r requirements.txt

Seems to run into cffi install/wheel setup problems.

Can't seem to find anyone else with this problem.

I'm not very familiar with these concepts.

I'll paste the error here, hopefully, someone has run into this before.

 

  note: This error originates from a subprocess, and is likely not a problem with pip.

  ERROR: Failed building wheel for cffi

  Running setup.py clean for cffi

Failed to build cffi

Installing collected packages: cffi, requests-futures, pymisp, cryptography, botocore, s3transfer, pyOpenSSL, awscli

  Attempting uninstall: cffi

    Found existing installation: cffi 1.15.1

    Uninstalling cffi-1.15.1:

      Successfully uninstalled cffi-1.15.1

  Running setup.py install for cffi ... error

  error: subprocess-exited-with-error

  × Running setup.py install for cffi did not run successfully.

  │ exit code: 1

  ╰─> [75 lines of output]

Copper Contributor

@LauriK000 What is your python version? Usually that means you’re on too high version, please try below 3.6.9 as specified above

Copper Contributor

@TechNashville has that fixed the issue for you?

Copper Contributor

@TechNashville This script works best on Ubuntu 18.04/ Python 3.6.9 btw 

Brass Contributor

Hey @camc ! The issue on my end actually turned out to be that MISP was filling my data disk. I just increased the size and it straightened out everything. So the method here also works for Ubuntu 20.04 / Latest Python3 release. :) 

 

Like another user stated here, it works best for me when running as 'Python3 script.py.'

 

I also have found that limiting the import to Sentinel to the most recent few days using the filter (details that I posted prior) helped out a lot, too. I suppose it's a lot to keep importing the ENTIRE db over and over. haha.

Copper Contributor

Hi All,

 

After running the script " sudo python3 script.py" on MISP Instance. I am getting below ouput.

Dilbag_Virk_0-1660855919683.png

 

 

I have followed all the steps and unable to get result in KQL ( Sentinel).

 

Any idea if i am missing something.

 

 

Copper Contributor

Hi, 

 

Anyone can tell me how to change the configuration to connect the same MISP server to connect multi SIEM? 

 

Thank you

Copper Contributor

@diaalfar you can't connect multiple SIEM. You can however use a workspace query in all of your workspaces referencing the workspace containing the CTI data, then just query as normal.

 

e.g. 

workspace("workspace-with-cti").ThreatIntelligenceIndicator

| where ...

Version history
Last update:
‎Nov 02 2021 05:55 PM
Updated by: