Forum Widgets
Latest Discussions
Cost-effective alternatives to control table for processed files in Azure Synapse
Hello, good morning.In Azure Synapse Analytics, I want to have a control table for the files that have already been processed by the bronze or silver layers. For this, I wanted to create a dedicated pool, but I see that at the minimum performance level it charges 1.51 USD per hour (as I show in the image), so I wanted to know what other more economical alternatives I have, since I will need to do inserts and updates to this control table and with a serverless option this is not possible.How to rigger an azure synapse pipeline after a file is dropped into an azure file-share.
I am currently looking for ideas to trigger an azure synapse pipeline after a file is dropped into an azure file-share. I feel that azure functions might be a suitable candidate to implement this. Azure synapse pipelines don't natively support this at the moment. Microsoft do offer a custom events trigger extension capability. However, so far I have found very little evidence demonstrating how to leverage this capability to trigger my synapse pipeline. Any assistance with approaches to solving this would be greatly appreciated. Thanks.
ADF Pipeline Data flow issue
I've created a data flow where the source is ADLS and the sink is an ADLS delta file. I attempted to run the data flow, but I encountered the following issue Job failed due to reason: com.microsoft.dataflow.Issues: DF-EXPR-200 - Function 'outputs' applies to only sink transformation with right saveOrder - [418 464 564 737],[207 319 418 464 564 737],EXE-0001, surrogateKey1 derive( ) ~> derivedColumn1,Dataflow cannot be analyzed as a graph,[62 207 319 418 464 564 737] DF-EXPR-200 - Function 'outputs' applies to only sink transformation with right saveOrder - [418 464 564 737],EXE-0001, surrogateKey1 derive( ) ~> derivedColumn1,Dataflow cannot be analyzed as a graph,[209 319 418 464 564 737]
Should I ingest AADNonInteractiveUserSignInLogs from Entra ID to a LAW
As the title says, I am interested in expert opinions on whether I should include the AADNonInteractiveUserSignInLogs from Entra ID in a LAW, as this table dwarfs the SignInLogs in terms of the amount of data (by a factor of 8x) and therefore creates higher costs. Secondly, I am curious if there are ways to reduce the amount of non-interactive SignInLogs that are generated in the first place.
Unable to retrieve query data using Log Analytics API
I have been trying to access Azure KQL data with the help of Log Analytics REST API, the connection is successful showing a 200 response but I am only getting the table headers and not getting any data in the table. Does anyone know how to resolve this? Code snippet: import requests import urllib3 from azure.identity import DefaultAzureCredential from datetime import datetime, timedelta, timezone import certifi import os os.environ["REQUESTS_CA_BUNDLE"] = certifi.where() verify_cert = certifi.where() credential = DefaultAzureCredential() # Set the start and end time for the query end_time = datetime.now(timezone.utc) start_time = end_time - timedelta(hours=6) # Set the query string query = ''' KubePodInventory | take 5 ''' # Set the workspace ID workspace_id = "XXXXXXXXXXXXXXXXXXXXXXXX" # Set the API endpoint api_endpoint = f"https://api.loganalytics.io/v1/workspaces/{workspace_id}/query" # Set the request payload payload = { "query": query, "timespan": f"{start_time.isoformat()}Z/{end_time.isoformat()}Z" } # Set the request headers headers = { "Content-Type": "application/json" } # Disable SSL certificate verification urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) # Authenticate the request using the Azure credential access_token = credential.get_token("https://api.loganalytics.io/.default").token headers["Authorization"] = f"Bearer {access_token}" # Send the POST request response = requests.post(api_endpoint, json=payload, headers=headers, verify=False) # Check the response status if response.status_code == 200: data = response.json() tables = data.get('tables', []) if tables: table = tables[0] # Assuming there is only one table returned columns = table.get('columns', []) rows = table.get('rows', []) if columns and rows: for row in rows: for i, column in enumerate(columns:( column_name = column['name'] column_type = column['type'] row_value = row[i] print(f"Column name: {column_name}, Data type: {column_type}, Value: {row_value}") else: print("Empty table or no data in table") else: print("No tables found in the response") else: print(f"Request failed with status code: {response.status_code}") print(f"Error message: {response.text}")Data archiving of delta table in Azure Databricks
Hi all, Currently I am researching on data archiving for delta table data on Azure platform as there is data retention policy within the company. I have studied the documentation from Databricks official (https://docs.databricks.com/en/optimizations/archive-delta.html) which is about archival support in Databricks. It said "If you enable this setting without having lifecycle policies set for your cloud object storage, Databricks still ignores files based on this specified threshold, but no data is archived." Therefore, I am thinking how to configure the lifecycle policy in azure storage account. I have read the documentation on Microsoft official (https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview) Let say the delta table data are stored in "test-container/sales" and there are lots of "part-xxxx.snappy.parquet" data file stored in that folder. Should I simply specify "tierToArchive", "daysAfterCreationGreaterThan: 1825", "prefixMatch: ["test-container/sales"]? However, I am worried that will this archive mechanism impact on normal delta table operation? Besides, I am worried that what if the parquet data file moved to archive tier contains both data created before 5 years and after 5 years, it is possible? Will it by chance move data earlier to archive tier before 5 years? Highly appreciate if someone could help me out with the questions above. Thanks in advance.
Understanding Data Ingestion for Log Analytics and Sentinel Workspace
I'm trying to understand how data ingestion works for both Log Analytics and Microsoft Sentinel. Every time we notice a spike in data ingestion costs for Log Analytics, we see a similar increase in Sentinel costs as well. It seems like data is being ingested into both workspaces, potentially doubling the ingestion and driving up our costs. Can someone explain if this is expected behavior, or if there's a way to optimize and avoid duplicate data ingestion between Log Analytics and Sentinel?
Understanding Data Ingestion for Log Analytics and Sentinel Workspace
I'm trying to understand how data ingestion works for both Log Analytics and Microsoft Sentinel. Every time we notice a spike in data ingestion costs for Log Analytics, we see a similar increase in Sentinel costs as well. It seems like data is being ingested into both workspaces, potentially doubling the ingestion and driving up our costs. Can someone explain if this is expected behavior, or if there's a way to optimize and avoid duplicate data ingestion between Log Analytics and Sentinel?Finetuning Sentinel alert for user added to Azure Active Directory Privileged Groups
I am here to ask for help with finetuning an alert in Sentinel. The alert is called "User added to Azure Active Directory Privileged Groups", and it's triggered when a user is added to a privileged group in Azure Active Directory. The problem is that this alert is also triggered when IAM team members activate their PIM roles for day-to-day activities. This is not a security incident, and it's causing us to get a lot of false positives. I'm trying to figure out a way to finetune the alert so that it's only triggered for actual security incidents. I am looking for ways or any ideas for finetuning the alert. to proceed with this further. If you have any suggestions, please let me know.
