Forum Discussion
Issue running DLT pipeline
I am new to Azure. I was trying to do a lab in Azure data engineering:Create a data pipeline with Delta Live tables. I followed all steps as mentioned in the lab instructions in Microsoft learn. But I got the following issue when I tried to start Delta live tables pipeline.
DataPlaneException: [DLT ERROR CODE: CLUSTER_LAUNCH_FAILURE.CLOUD_PROVIDER_FAILURE] Failed to launch pipeline cluster 0801-165638-xpgr89hv with termination code AZURE_QUOTA_EXCEEDED_EXCEPTION and termination type CLIENT_ERROR: Encountered Quota Exhaustion issue in your account: azure_error_code: QuotaExceeded, azure_error_message: Operation could not be completed as it
results in exceeding approved Total Regional Cores ..
Could anybody please help me out? What can I do? I am clueless.
4 Replies
- The error you're encountering indicates that your Azure account has exceeded the quota for the total number of regional cores. This is a common issue when you reach the limits set by Azure for your subscription.
- What sort of Azure subscription are you using?
The error basically says that you do not have enough CPU cores available in the quota in your subscription and therefore cannot start the compute cluster that is needed to run your pipeline
See https://learn.microsoft.com/en-us/answers/questions/1339289/my-pipeline-is-failing-in-resource-awaiting-and-ge for possible solutionJulian_Sharp My azure subscription is pay as you go. I wanted to do the lab,Create a data pipeline with Delta Live tables and configured the cluster as it mentioned which was:
- Cluster name: User Name’s cluster (the default cluster name)
- Policy: Unrestricted
- Cluster mode: Single Node
- Access mode: Single user (with your user account selected)
- Databricks runtime version: 13.3 LTS (Spark 3.4.1, Scala 2.12) or later
- Use Photon Acceleration: Selected
- Node type: Standard_DS3_v2
- Terminate after 20 minutes of inactivitygot the issue as before and then tired to do the solution with:
but got the issue even while creating the cluster and gave me the issue as:
Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardDDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus2, Current Limit: 0, Current Usage: 0, Additional Required: 8, (Minimum) New Limit Required: 8. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%222469364a-b4bd-4787-9d36-14e4e57a572b%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus2%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:8,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests
I have requested the increase in quota from 0 to 50. So, waiting for their response. Will try after the quota increase and inform you what happens?
- Yes, I was going to suggest you request an increase in your quota if you had more than a free subscription
In the meantime just reduce the number of cores in the size of the VM you have selected as it probably won't make much difference in how long the experiments will take to run given the size of data in the labs or maybe select a different region and see if yuo have more quota there.
