What's New: CrowdStrike Falcon Data Replicator V2 Data Connector is now Generally Available!

[PrateekTaneja]

The CrowdStrike Falcon Data replicator V2 Data connector is now Generally Available as a part of the CrowdStrike Falcon Endpoint Protection solution in Microsoft Sentinel Content Hub. The connector leverages an Azure Function based backend to poll and ingest CrowdStrike FDR logs at scale. Some of the advantages this new V2 data connector offers are:

1. Improved scaling as per data volume - keeping the performance of ingestion high.
2. More data ingestion with the Consumption plan, hence, optimizing cost.
3. Ingestion-time Normalization to the ASIM data model, enabling customers to use various normalized solutions and associated content (Analytics, hunting, workbooks).
4. Query time parsing is faster because of data split into multiple tables based on event category (like Network, Authentication, File, DNS etc.).
5. CrowdStrike secondary data (like appinfo, assetinfo, userinfo etc) can also be ingested.
6. It supports ingestion of raw logs in addition to normalized logs (for compliance purposes if needed).

High level design of this connector

The CrowdStrike logs collected from AWS S3 bucket will be stored in relevant normalized ASIM tables by default and if one opts for storing the raw logs then it will get stored in CrowdStrike custom tables.

Getting started

Installing the solution from Content Hub will deploy all 3 data connectors in data connector gallery (if one already has the CS solution installed, they will get an option to update the solution this add the new V2 data connector along with the existing ones in the data collector gallery)

Once the solution is installed click on manage solution, select the V2 data connector and click open data connector page on the right-side panel.

Note: Ensure all the listed prerequisites are taken into consideration.

The ingestion requires an Azure Function to be deployed. Click on the Deploy to Azure button, or alternatively, follow the manual steps to deploy.

For configuring the Azure Function App parameters,

1. Set the right subscription, region, workspace and select the appropriate number of connector instances based on the number of logs expected to be ingested. Click Next.
2. Specify the CS AWS details and Azure AD application details.
3. Set the EPS (events per seconds) based on the expected data volume

If required, select the check box to ingest secondary data and ingest raw copy of normalized data

Click on Review and Create for the connector to get deployed. Once deployed successfully, the connector will start polling and ingesting to the Microsoft Sentinel workspace.

Here are some FAQs:

1. How much volume can Sentinel CrowdStrike Falcon Data Replicator V2 data connector process?
   1. CrowdStrike Falcon Data Replicator V2 is a cloud-based solution that can scale according to your data volume. You need to choose the appropriate number and type of connector instances to balance the cost and performance based on your needs.
2. I do not know what the EPS that CrowdStrike generates? What value should I enter during connector setup?
   1. To estimate your EPS or data size, you can use the default settings (40,000 EPS) as a starting point. After a couple of days, compare the time stamps of the data you ingested and the data that was generated. You can also calculate EPS by dividing the daily event count by 86,400 seconds (1 day). Make sure you use the timestamps coming from source for the daily event count. If there is a significant gap between the ingestion time and the generation time of the data, you may need to add another instance of data connector or switch to Premium. This way, you can adjust your settings based on your actual EPS.
3. If I know only data size in GBs / TBs at my source (CrowdStrike), how to estimate EPS?
   1. While data in un-compressed format, we can estimate 100 MB = 60,000 events (approximately)
   2. For compressed data, we can estimate 100 MB = 360, 000 events (approximately)
4. How do I find AWS credentials that are required during connector deployment?
   1. Configure FDR in CrowdStrike - You must contact the CrowdStrike support team https://supportportal.crowdstrike.com/  to enable CrowdStrike FDR.
      1.       Once CrowdStrike FDR is enabled, from the CrowdStrike console, navigate to Support --> API Clients and Keys.
      2.       You need to Create new credentials to copy the AWS Access Key ID, AWS Secret Access Key, SQS Queue URL and AWS Region.
   2. Customers do not have Falcon Data Replicator (FDR) configured. Can I still use this data connector?
      1. No. This data connector is dependent on FDR, without enabling we cannot use this data connector. You must contact the CrowdStrike support team https://supportportal.crowdstrike.com/  to enable CrowdStrike FDR.
   3. We have multiple data connectors for CrowdStrike Falcon Endpoint Protection and CrowdStrike Falcon Data Replication, and CrowdStrike Falcon Data Replication V2 - what is the difference?
      1. All 3 data connectors are targeted to ingest the same data into Sentinel. So, use only 1 of below data connectors to ingest the CrowdStrike data. Otherwise, you will end up ingesting duplicated data.
         1.       CrowdStrike Falcon Endpoint Protection – This data connector helps to ingest CrowdStrike data using MMA (Microsoft Monitoring Agent) / AMA agent. Use this data connector if you want to use agent-based data ingestion.
         2.       CrowdStrike Falcon Data Replication – This data connector fetches data from AWS S3 into Custom Log Analytics tables.

•       CrowdStrike Falcon Data Replication V2 – This is the latest one. This data connector fetches data from AWS S3 and normalizes at ingestion time.

7. Data is getting ingested successfully using CrowdStrike Falcon Data Replication V2, what should I do next? [Coming Soon]
   1. Deploy and enable content from domain solutions – Since data is getting normalized during ingestion time itself and getting ingested into normalized tables, you must take benefit of all ASIM based content like ASIM query time parsers ( _ASim_NetworkSession, _ASim_Dns, _ASim_Websession, _ASim_Audit etc) and ASIM domain solutions like network session essentials, web session essentials, dns essentials etc.
8. Why are there few tables built-in and few are custom tables?
   1. Built-in tables holds the data in normalized format using ASIM schema, and we ingest data into these tables by transforming into ASIM during the ingestion time itself.
   2. If any event that does not fit into any of the ASIM schemas. Such event types go to separate table called – CrowdStrike_Additional_Events_CL
9. I am ingesting data into normalized tables, do I still need to collect raw data into separate tables?
   1. No, unless it is required for special purposes like compliance, audit, monitoring, etc.
   2. Collect data in raw format only if necessary as it incurs additional cost.
10. I only want to collect data in original format and don’t want to ingest into Normalized tables? Is this possible?
    1. Currently ingesting data only in raw format is not possible.
    2. By default, this data connector ingests data in normalized format (wherever applicable). The rest of the data will be ingested in raw format into a custom table - CrowdStrike_Additional_Events_CL
11. What is data retention set for raw data?
    1. We store raw copy into LA Basic tier, which costs less for customers.
    2. We set default retention (8 days) and customers need to change if they need to modify the retention.
    3. The above things can be modified id required. Please refer to https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-configure?tabs=portal-1#set-a-...
12. What is the difference between function app consumption and premium plans?
    1. You can reference here - https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#overview-of-plans
13. I am using previous versions of CrowdStrike Falcon Data Replicator data connector. How do I migrate to CrowdStrike Falcon Data Replicator V2?
    1. If you want to start using the new data connector (CrowdStrike Falcon Data Replicator V2), first you need to stop data ingestion with old data connector (CrowdStrike Falcon Data Replicator). This can be done by stopping the app that is being used by CrowdStrike Data Replicator.
    2. Navigate to Function App à Search and open for crowdstrike******  à Overview à Stop.

1. [Optional – Can be later] You can delete the respective function app and dependent resources later for cleanup. This can be done by Navigate to resource group à Search for Resources by using the name same as function app à Select all that are matching à Delete.

2. Deploy and configure new data connector CrowdStrike Falcon Data Replicator V2.
3. Once you have the data ingestion starts working using new data connector, you must see data in 1 or more of following tables. This can be verified by opening new data connector à Data Received section.

14. Using old data connector data ingested into CrowdstrikeReplicatorLogs_CL table, if I continue with the old data connector am I going to miss something?
    1. It is highly recommended to migrate to the new data connector CrowdStrike Falcon Data Replicator V2, so that you can see all the benefits that are mentioned earlier.
    2. Using the old CrowdStrike FDR data connector will not break anything. No retirement plan has been decided for the old data connector, so it will continue functioning.