Ingesting .CSV log files from Azure Blob Storage into Microsoft Sentinel
Overview: Organizations generate vast amounts of log data from various applications, services, and systems. These logs are often stored in .CSV (Comma-Separated Values) format in Azure Blob Storage, a scalable cloud-based storage solution. To enhance security monitoring, compliance, and threat detection, it is important to bring this log data into a centralized security tool like Microsoft Sentinel. The main goal is to automatically collect and analyze .CSV log files stored in Azure Blob Storage using Sentinel’s advanced analytics and automation capabilities. This enables better visibility into security events and helps in proactive threat management. Benefits: Flexible Log Ingestion via logic app: Allows ingestion of logs from systems without built-in Sentinel connectors, including custom, third-party, or legacy systems. Uses Existing Storage Workflows: Reuses Azure Blob Storage where logs are already being saved, with no need to change current export methods. Structured and Clean Data Format: .CSV files offer a structured format that makes mapping and parsing data into Sentinel efficient and reliable. Enables Custom Analysis: Once in Sentinel, the data can be queried using Kusto Query Language (KQL) for in-depth analysis and reporting. Operational Efficiency: Reduces manual efforts in collecting, uploading, or processing logs. Saves time for IT and security teams by automating the data pipeline. Improves Threat Visibility: Ingested data is available in real-time. Dashboards and visualizations make it easy to understand what's happening. Pre-requisites: Log Analytics Workspace A configured workspace to receive and analyze the ingested data. Blob Storage Path The exact location in Azure Blob Storage where the CSV log files are stored. Required Roles and Permissions Microsoft Sentinel Contributor– to manage Sentinel resources. Logic App Contributor– to create and manage automation workflows. Access to the Storage Account– to read and retrieve log files from Blob Storage. Implementation Steps: Configure the Logic App trigger to run whenever a new blob is added or an existing one is modified. Select the storage account and container details, then configure the recurrence based on how frequently data is uploaded to the storage account. Choose the authentication type to connect with storage account. CSV Retrieval: Use the Logic App action to retrieve the CSV blob content by specifying the exact file path of the container. CSV Parsing: Use built-in Logic App actions along with regex to parse the CSV content. Apply the Composeaction to split the file contents by new lines, converting them into an array for structured processing. Here is the expression used in SplitLines compose action: split(body('Get_blob_content_(V2)'),decodeUriComponent('%0D%0A')) Follow the below MS Doc to write expressions: Removing last(empty) line from previous output using another compose action as shown below, take(outputs('SplitLines'),add(length(outputs('SplitLines')),-1)) Separating field names using compose action: split(first(outputs('SplitLines')), ',') Column Mapping: Repeat the required expression using the Select action to map each column from the CSV file to its corresponding field in the structured output. **From**: **`skip(outputs('RemoveLastLine'), 1)`** **Map:** **`outputs('SplitFieldName')[0]`** **`split(item(), ',')?[0]`** **`outputs('SplitFieldName')[1]`** **`split(item(), ',')?[1]`** Data Ingestion to Sentinel: Leveraging the Microsoft Sentinel connector to ingest the parsed data into the appropriate table. The connection to be configured using the workspace ID, shared key, and target table name. Key Highlights: The Logic App is triggered whenever a file is added or modified in the Blob container. The CSV content is parsed within the Logic App before being ingested into Sentinel. Leveraged the Microsoft Sentinel connector to ingest the parsed data into Sentinel. To support dynamic updates, we recommended overwriting the existing CSV file in the storage account. Outcome: Log Visibility in Sentinel Workspace: Once the Logic App is triggered, the custom table will be created automatically in Microsoft Sentinel, and logs can be viewed by running a KQL query in the Sentinel workspace. Conclusion: Ingesting .CSV log files from Azure Blob Storage into Microsoft Sentinel is a powerful way to centralize and automate the organization’s security monitoring. It enhances visibility, supports compliance, and empowers security teams with timely insights and alerts.
Recent Logic Apps Failures with Defender ATP Steps – "TimeGenerated" No Longer Recognized
Hi everyone, I’ve recently encountered an issue with Logic Apps failing on Defender ATP steps. Requests containing the TimeGenerated parameter no longer work—the column seems to be unrecognized. My code hasn’t changed at all, and the same queries run successfully in Defender 365’s Advanced Hunting. For example, this basic KQL query: DeviceLogonEvents | where TimeGenerated >= ago(30d) | where LogonType != "Local" | where DeviceName !contains ".fr" | where DeviceName !contains "shared-" | where DeviceName !contains "gdc-" | where DeviceName !contains "mon-" | distinct DeviceName Now throws the error: Failed to resolve column or scalar expression named 'TimeGenerated'. Fix semantic errors in your query. Removing TimeGenerated makes the query work again, but this isn’t a viable solution. Notably, the identical query still functions in Defender 365’s Advanced Hunting UI. This issue started affecting a Logic App that runs weekly—it worked on May 11th but failed on May 18th. Questions: Has there been a recent schema change or deprecation of TimeGenerated in Defender ATP's KQL for Logic Apps? Is there an alternative column or syntax we should use now? Are others experiencing this? Any insights or workarounds would be greatly appreciated!
Adding users to an AD group with Azure Functions/Logic Apps
I want to add users to an Entra ID/Azure AD group. The list of users will be retrieved from a REST API call with Azure Functions, and then saved into a database, probably Azure SQL. I'm planning on then using Azure Logic Apps to connect the database to the AD group. How can I make the script run every time the REST API changes? Can I add users to the AD group from SQL? Is there a better way to go about this?For loop getting data from the API and sending all at one to the email
Hi there, Learning Azure Logic Apps for my job. I have a small azure logic app that needs to export API data with and send it to the email. So far all the steps except adding the output body from the HTTP get request to the variable. Yes, seems that it's just rewriting the variable. But how can I append to a variable without rewriting it? Any advice? Here's the actual build idea. Iterating through profile ID's and getting the output for every ID, then I need to be able to push all of that to the email. Next step for me would be to prettify all this, but that's a bit outside this question. Thanks and much appreciated!
Call SAP OData Service in Logic App through On-Premise Data Gateway
Hello, I want to call an Odata Service of my SAP system. The service is only available in the private network, therefore the call only works via an On Premise Data Gateway. This gateway is already set up and works in the Azure Portal. Using a Postman Collection I have built a Custom Connector which I call in my Logic App. The connector is connected via the gateway to a computer which is logged on to the VPN. My problem is that I can't provide the authentication data for the logon to the SAP system and therefore for the call of the OData service. Is there a possibility to include this authentication data in the request? I have already been able to connect successfully to the SAP system via that gateway, but only to call RFC modules, for example. Is there a way to do this with an OData service? What is the best way for calling an on-premise SAP OData Service in a Logic App? Or is there maybe a way to send an Http request via the On Premise Data Gateway? Thank you!
How to start a call within the existing MS Teams channel using azure logic app
Hi All, I have a requirement where our azure Logic App will create a new MS Teams, add members into MS Teams and add channel into same MS Teams. Now my next requirement is I want to start a call immediately in the same channel which we have created using Logic App and also this should be done in the same logic app job which we have build for creating MS Teams and channel. As I know there is no separate connector available in Azure logic app to achieve this functionality. So, I dig a little bit and I found below approach. 1. Register a new application using the Azure portal. 2. Get the token from application. 3. Use below rest API method. https://graph.microsoft.com/v1.0/me/onlineMeetings 4. Provide details in the body like start time, end time etc. 5. Call above rest API using azure logic app. So, from the above steps my question is, in the body section where we have included start time, end time etc. can we include same participants which we have added in MS Teams while creating azure logic app. If yes, then how? If it is possible then the flow of my Azure Logic App will be as follow. Create MS Teams --> Add Members into Teams --> Create Channel --> Post the messages into channel --> Call the Graph API (https://graph.microsoft.com/v1.0/me/onlineMeetings) Note:- As soon as Azure Logic App execution is completed it should start a call immediately.
Connect to SharePoint Online using a Logic App and Key Vault
I'm looking for info on how to connect to SharePoint Online using a Logic App and Key Vault. Also, what authentication method should be used? Currently, we are using service accounts to connect to SPO, but the accounts require password updates every quarter and maintaining this is getting out of control.Azure SharePoint Logic App Connector Security
Using the SharePoint Logic App connector, how can the Azure Key Vault be leveraged in place of a hard-coding AD account? Currently, we are using a dedicated AD account to authenticate with SharePoint, and our password policy requires we update the password every X months. As the usage of Azure and SPO grows, this model is quickly becoming unmanageable.Wrong MDATP Logic App Connector Auth. endpoint for USgov
I'm trying to create a logic app that will trigger when a new WDATP alert occurs inside of a USgov region using the MDATP connector in the logic app designer. When I click the "Sign in" button it takes me to the authentication URL at https://login.microsoftonline.com/ which is not the proper authentication endpoint for USgov (it should redirect me to https://login.microsoftonline.us) This causes an error response letting me know that I'm making a request to a public endpoint instead of the government endpoint, and that the application must send the user to the right endpoint. I've spent hours looking for ways to change the authentication endpoint to the USgov one in the Microsoft Defender ATP logic app connector and I'm out of ideas. Has anyone encountered this issue and was able to edit the connector's request? or found a workaround? I'd love to hear from someone, thank you!
