Forum Discussion
Azure Synapse - Serverless Sql Pool block querys when start to sync with hive metastore using delta
i'm using a lake db with delta format using pyspark, the problem that i'm having is: when new metadata is saved in hive, serverless sql pool run a internal delta query to sync the new metadata, ok, but sometimes this query freezes and do not stop more, blocking the entyre serverless sql pool connection, i don't now why this is happening, i have tryed create another lakes, drop current lakes, change configs, but nothing solve the problem. Sometimes everything works perfectly, sometimes one sync internal delta query freezes in a random table and block another querys until kill the process
EDIT:
This problem was not related directly with serverless pool hive sync, this was only a casualitty of a bigger problem with serverless sql pool connection and data lake, we are investigating better
- Hi ismaelhenzel,
It sounds like you are experiencing an issue with Azure Synapse Serverless SQL Pool query performance when syncing with Hive metastore using Delta. Here are some suggestions that might help:
Check the resource usage of your Synapse Serverless SQL Pool to ensure that it has enough resources to handle the load. You can scale up the pool if necessary.
Check if there are any specific tables that are causing the issue. If so, try optimizing the schema or partitioning the table.
Check if there are any slow-running queries that might be causing the issue. You can use the query store feature in Synapse to identify and optimize slow-running queries.
Ensure that your Delta Lake version is compatible with Synapse Serverless SQL Pool.
Consider using a dedicated SQL Pool instead of a Serverless SQL Pool if you require consistent performance.
Contact Microsoft Azure support for further assistance if the issue persists.
I hope this helps. Let me know if you have any other questions.
Kind regards,
Luke Madden
"Simplifying Tech, Empowering you."
- Hi ismaelhenzel,
It sounds like you are experiencing an issue with Azure Synapse Serverless SQL Pool query performance when syncing with Hive metastore using Delta. Here are some suggestions that might help:
Check the resource usage of your Synapse Serverless SQL Pool to ensure that it has enough resources to handle the load. You can scale up the pool if necessary.
Check if there are any specific tables that are causing the issue. If so, try optimizing the schema or partitioning the table.
Check if there are any slow-running queries that might be causing the issue. You can use the query store feature in Synapse to identify and optimize slow-running queries.
Ensure that your Delta Lake version is compatible with Synapse Serverless SQL Pool.
Consider using a dedicated SQL Pool instead of a Serverless SQL Pool if you require consistent performance.
Contact Microsoft Azure support for further assistance if the issue persists.
I hope this helps. Let me know if you have any other questions.
Kind regards,
Luke Madden
"Simplifying Tech, Empowering you."
