Forum Discussion
Issue running DLT pipeline
I am new to Azure. I was trying to do a lab in Azure data engineering:Create a data pipeline with Delta Live tables. I followed all steps as mentioned in the lab instructions in Microsoft learn. But ...
Julian_Sharp My azure subscription is pay as you go. I wanted to do the lab,Create a data pipeline with Delta Live tables and configured the cluster as it mentioned which was:
- Cluster name: User Name’s cluster (the default cluster name)
- Policy: Unrestricted
- Cluster mode: Single Node
- Access mode: Single user (with your user account selected)
- Databricks runtime version: 13.3 LTS (Spark 3.4.1, Scala 2.12) or later
- Use Photon Acceleration: Selected
- Node type: Standard_DS3_v2
- Terminate after 20 minutes of inactivitygot the issue as before and then tired to do the solution with:
but got the issue even while creating the cluster and gave me the issue as:
Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardDDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus2, Current Limit: 0, Current Usage: 0, Additional Required: 8, (Minimum) New Limit Required: 8. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%222469364a-b4bd-4787-9d36-14e4e57a572b%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus2%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:8,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests
I have requested the increase in quota from 0 to 50. So, waiting for their response. Will try after the quota increase and inform you what happens?
Yes, I was going to suggest you request an increase in your quota if you had more than a free subscription
In the meantime just reduce the number of cores in the size of the VM you have selected as it probably won't make much difference in how long the experiments will take to run given the size of data in the labs or maybe select a different region and see if yuo have more quota there.
In the meantime just reduce the number of cores in the size of the VM you have selected as it probably won't make much difference in how long the experiments will take to run given the size of data in the labs or maybe select a different region and see if yuo have more quota there.