Control Azure Data Lake costs using Log Analytics to create service alerts
Azure Data Lake customers use the Data Lake Store and Data Lake Analytics to store and run complex analytics on massive amounts of data. However, it is challenging to manage costs, keep up-to-date with activity in the accounts, and proactively know when usage thresholds are nearing certain limits. Using Log Analytics and Azure Data Lake we can address these challenges and know when the costs are increasing or when certain activities take place. https://azurecomcdn.azureedge.net/mediahandler/acomblog/media/Default/blog/eaeb8328-7731-4020-8c74-f4e694bd0bef.png In this post, you will learn how to use Log Analytics with your Data Lake accounts to create alerts that can notify you of Data Lake activity events and when certain usage thresholds are reached. It is easy to get started! Read more about it in the https://azure.microsoft.com/en-us/blog/control-azure-data-lake-costs-using-log-analytics-to-create-service-alerts/.Using Azure Analysis Services with Azure Data Lake Storage
Support for Azure Data Lake Store (ADLS) is now available in Azure Analysis Services and in SQL Server Data Tools (SSDT). Now you can augment your big data analytics workloads with rich interactive analysis for selected data subsets at the speed of thought! Business users can consume Azure Analysis Services models in Microsoft Power BI, Microsoft Office Excel, and Microsoft SQL Server Reporting Services. Azure Data Lake Analytics (ADLA) can be used to run U-SQL batch jobs directly against the source data, such as to generate targeted output files that Azure Analysis Services can import with less overhead. Azure Data Lake Analytics (ADLA) can process massive volumes of data extremely quickly. Exporting approximately 2.8 billion rows of TPC-DS store sales data (~500 GB) into a CSV file took less than 7 minutes and importing the full 1 TB set of source data into Azure Analysis Services by using the Azure Data Lake connector took less than 6 hours. These results highlight Azure Data Lake as an attractive big-data backend for Azure Analysis Services. Read about it in the https://azure.microsoft.com/en-us/blog/using-azure-analysis-services-with-azure-data-lake-storage/.Run Hortonworks clusters and easily access Azure Data Lake
Enterprise customers love Hortonworks for running Apache Hive, Apache Spark and other Apache Hadoop workloads. They also love the value that Azure Data Lake Store (https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-overview) provides, like high throughput access to cloud data of any size, sharing easily and securely with its true hierarchical file system, Posix ACLs, along with Role-based Access Control (RBAC), and encryption-at-rest. Azure HDInsight managed workloads – which offers built-in integration with and access to ADLS – vastly simplifies the management of enterprise clusters for many enterprises. Customers have a choice, and some Hortonworks customers choose to customize and manage their own clusters deployed directly on Azure cloud infrastructure, and those deployments need direct access ADLS. Read about it in the https://azure.microsoft.com/en-us/blog/run-hortonworks-clusters-and-easily-access-azure-data-lake/.Microsoft Azure Data Lake Storage in Storage Explorer – public preview
Providing a rich GUI for Azure Data Lake Storage resources management has been a top customer ask for a long time, we are thrilled to announce the public preview for supporting Azure Data Lake Storage (ADLS) in the Azure Storage Explorer (ASE). With the release of ADLS resources in ASE, you can freely navigate ADLS resources, you can upload and download folders and files, you can copy and paste files across folders or ADLS accounts and you can easily perform CRUD operations for your folders and files. Azure Storage Explorer not only offers a traditional desktop explorer GUI for dragging, uploading, downloading, copying and moving your ADLS folders and files, but also provides a unified developer experiences of displaying file properties, viewing folder statistics and adding quick access. With this extension you are now able to browse ADLS resources along-side existing experiences for Azure Blobs, tables, files, queues and Cosmos DB in ASE. Read about it in the https://azure.microsoft.com/en-us/blog/microsoft-azure-data-lake-storage-adls-in-storage-explorer-public-preview/.Azure Data Lake tools integrates with VSCode Data Lake Explorer and Azure Account
If you are a data scientist and want to explore the data and understand what is being saved and what the hierarchy of the folder is, please try Data Lake Explorer in VSCode ADL Tools. If you are a developer and look for easier navigation inside the ADLS, please use Data Lake Explorer in VSCode ADL Tools. The VSCode Data Lake Explorer enhances your Azure login experiences, empowers you to manage your ADLA metadata in a tree like hierarchical way and enables easier file exploration for ADLS resources under your Azure subscriptions. You can also preview, delete, download, and upload files through contextual menu. With the integration of VSCode explorer, you can choose your preferred way to manage your U-SQL databases and your ADLS storage accounts in addition to the existing ADLA and ADLS commands. If you have difficulties to login to Azure and look for simpler sign-in processes, the Azure Data Lake Tools integration with VSCode Azure account enables auto sign in and greatly enhance the integration with Azure experiences. If you are an Azure multi-tenant user, the integration with Azure account unblocks you and empowers you to navigate your Azure subscription resources across tenants. Read about it in the https://azure.microsoft.com/en-us/blog/azure-analysis-services-features-on-azure-friday/.Announcing Public Preview of HDInsight HBase on Azure Data Lake Store
On November 21, Microsoft announced the general availability of Azure Data Lake Store. Azure Data Lake Store is a hyperscale cloud storage for big data analytics built to the open Hadoop File System (HDFS) standard. Azure Data Lake Store provides enterprise grade security, including SSL and encryption at rest by default along with role based access control. Today we are excited to announce the public preview of HDInsight HBase on Azure Data Lake Store. Customers can harness the power of a columnar NoSQL distributed database with the proven performance and infinite scalability of Azure Data Lake Store. Azure Data Lake Store has no limits to capacity so customers will never need to worry about the limitations of their storage system. Furthermore, customers can store all their data and do all their analytics in one single storage account. Read about it on the https://azure.microsoft.com/en-us/blog/announcing-public-preview-of-hdinsight-hbase-on-azure-data-lake-store/.Azure Data Lake Tools for VSCode supports Azure blob storage integration
We are pleased to announce the integration of VSCode explorer with Azure blob storage. If you are a data scientist and want to explore the data in your Azure blob storage, please try the Data Lake Explorer blob storage integration. If you are a developer and want to access and manage your Azure blob storage files, please try the Data Lake Explorer blob storage integration. The Data Lake Explorer allows you easily navigate to your blob storage, access and manage your blob container, folder and files. Read about it in the https://azure.microsoft.com/en-us/blog/azure-data-lake-tools-for-vscode-supports-azure-blob-storage-integration/.
Not able to execute "az dls fs upload xxx" through java code
I am trying to execute " az dls fs upload --account XXX--source-path "/local/xyz.txt" --destination-path "/temp/folder/" " through java code using Process p=Runtime.getRuntime().exec(command) ; But its not copy file to datalake. Please help me to figure out this. or is there any other way to do this.Cloudera now supports Azure Data Lake Store
With the release of Cloudera Enterprise Data Hub 5.11, you can now run Spark, Hive, and MapReduce workloads in a Cloudera cluster on Azure Data Lake Store (ADLS). Running on ADLS has the following benefits: Grow or shrink a cluster independent of the size of the data. Data persists independently as you spin up or tear down a cluster. Other clusters and compute engines, such as Azure Data Lake Analytics or Azure SQL Data Warehouse, can execute workload on the same data. Enable role-based access controls integrated with Azure Active Directory and authorize users and groups with fine-grained POSIX-based ACLs. Cloud HDFS with performance optimized for analytics workload, supporting reading and writing hundreds of terabytes of data concurrently. No limits on account size or individual file size. Data is encrypted at rest by default using service-managed or customer-managed keys in Azure Key Vault, and is encrypted with SSL while in transit. High data durability at lower cost as data replication is managed by Data Lake Store and exposed from HDFS compatible interface rather than having to replicate data both in HDFS and at the cloud storage infrastructure level. Read about it on the https://azure.microsoft.com/en-us/blog/cloudera-adls/.Online training for Azure Data Lake
We are pleased to announce the availability of new, free online training for Azure Data Lake. We’ve designed this training to get developers ramped up fast. It covers all the topics a developer needs to know to start being productive with big data and how to address the challenges of authoring, debugging, and optimizing at scale. Read more about it in the https://azure.microsoft.com/en-us/blog/online-training-for-azure-data-lake/
