As customers continue to standardize on data lakes and the Lakehouse architecture, users expect to be able to query the data in their data lake using SQL. In fact, approximately 41% of all code executed on Azure Databricks is SQL. The SQL Analytics service in Azure Databricks was created to provide SQL users with a familiar SQL-editor experience as well as provide optimized BI connections for querying and analyzing data in the data lake.
SQL Analytics Key Features
Below are some of the key features in the SQL Analytics service in Azure Databricks:
- The first key feature to highlight is the Query Editor. This editor provides a familiar experience (vs. the traditional notebook experience in Azure Databricks) where users can explore their databases, write SQL queries with intelligent auto-complete, and view query output in either a tabular display or in a rich set of visualizations.
- Users can turn the queries and visualizations they create in the query editor into Dashboards. Dashboards can include content from several different queries, and also allow for basic text/markdown cells. Once created, dashboard data can be manually refreshed or refreshed on a schedule.
- In addition to dashboards, queries can be scheduled, and alerts can be created to notify users when a field in the scheduled query meets a certain threshold. Alerts can even be created for multiple columns.
- SQL Analytics also has a catalog for dashboards and queries. This catalog allows users to save their queries and dashboards and share them with other users. Users can also add tags to their saved queries and dashboards to make them easier to search, and they can even use Favorites to mark frequently used queries and dashboards.
- The SQL Endpoint in the SQL Analytics service also provides easy connectivity to other BI and SQL tools via ODBC/JDBC connections. Tools such as Power BI can connect using the native Azure Databricks connector and take advantage of faster, more efficient ODBC/JDBC drivers.
- Another exciting feature in the SQL Analytics service is the ability to see Query History details. On the History page, users and admins can see details about all the queries that have been run. This includes specifics such as the query itself, who ran the query, what endpoint did it run on, was it successful, the duration, how much time was spent compiling-executing-result-fetching, rows read/returned, files scanned, bytes read from cache, and many other details.
- The final key feature to look at in the SQL Analytics service is the compute engine. SQL Analytics uses the same Delta Engine found in the rest of Azure Databricks. This means a single, consistent set of APIs and functions across the entire workspace. The SQL Analytics service goes one step further by also making use of the Photon-powered Delta Engine. This vectorized engine was purpose built for doing SQL and data frame operations while maintaining the same single, consistent set of APIs and functions currently used across the entire workspace.
For more information about the SQL Analytics service in Azure Databricks, check out the docs page and the Databricks launch blog. The SQL Analytics service is currently in Public Preview. Contact your Azure Databricks representative to request access. Get started with Azure Databricks by joining an Azure Databricks workshop.