Understanding Azure storage redundancy offerings
In this post we'll go over major Azure infrastructure terminology aiming to get more clarity on how various high availability and disaster recovery offerings within Azure storage work and then segway to discussing the redundancy options in more detail. Read away!Announcing the Next generation Azure Data Box Devices
Microsoft Azure Data Box offline data transfer solution allows you to send petabytes of data into Azure Storage in a quick, inexpensive, and reliable manner. The secure data transfer is accelerated by hardware transfer devices that enable offline data ingestion to Azure. Our customers use the Data Box family to move petabytes-scale data into Azure for backup, archival, data analytics, media and entertainment, training, and different workload migrations etc. We continue to get requests about moving truly massive amounts of data in a secure, simple and quick manner. We’ve heard you and to address your needs, we’ve designed a new, enhanced product to meet your data transfer needs. About the latest innovation in Azure Data Box Family Today, we’re excited to announce the preview of Azure Data Box 120 and Azure Data Box 525, our next-generation compact, NVMe-based Data Box devices. The new offerings reflect insights gained from working with our customers over the years and understanding their evolving data transfer needs. These new devices incorporate several improvements to accelerate offline data transfers to Azure, including: Fast copy - Built with NVMe drives for high-speed transfers and improved reliability and support for faster network connections Easy to use - larger capacity offering (525 TB) in a compact form-factor for easy handling Resilient - Ruggedized devices built to withstand rough conditions during transport Secure - Enhanced physical, hardware and software security features Broader availability – Presence in more Azure regions, meeting local compliance standards and regulations What’s new? Improved Speed & Efficiency NVMe devices offer faster data transfer rates, with copy speeds up to 7 GBps via SMB Direct on RDMA (100-GbE) for medium to large files, a 10x improvement in data transfer speeds as compared to previous generation devices. High-speed transfers to Azure with data upload up to 5x faster for medium to large files, reducing the lead time for your data to become accessible in the Azure cloud. Improved networking with support for up to 100 GbE connections, as compared to 10 GbE on the older generation of devices. Two options with usable capacity of 120 TB and 525 TB in a compact form factor meeting OSHA requirements. Devices ship the next day air in most regions. Learn more about the performance improvements on Data Box 120 and Data Box 525. Enhanced Security The next generation devices come with several new physical, hardware and software security enhancements. This is in addition to the built in Azure security baseline for Data Box and Data Box service security measures currently supported by the service. Secure boot functionality with hardware root of trust and Trusted Platform Module (TPM) 2.0. Custom tamper-proof screws and built-in intrusion detection system to detect unauthorized device access. AES 256-bit BitLocker software encryption for data at rest is currently available. Hardware encryption via the RAID controller, which will be enabled by default on these devices, is coming soon. Furthermore, once available, customers can enable double encryption through both software and hardware encryption to meet their sensitive data transfer requirements. These ISTA 6A compliant devices are built to withstand rough conditions during shipment while keeping both the device and your data safe and intact. Learn more about the enhanced security features on Data Box 120 and Data Box 525. Broader Azure region coverage Recurring request from our customers has been for wider availability of our higher-capacity device to ease large migrations. We’re happy to share Data Box 525 will be available across most Azure regions where the Data Box service is currently live. This marks a significant improvement in availability of a large-capacity device as compared to the current Data Box Heavy. What our customers have to say For the last several months, we’ve been working directly with our customers of all industries and sizes to leverage the next generation devices for their data migration needs. Customers love the larger capacity with form-factor familiarity, seamless set up and faster copy. “This new offering brings significant advantages, particularly by simplifying our internal processes. With deployments ranging from hundreds of terabytes to even petabytes, we previously relied on multiple regular Data Box devices—or occasionally Data Box Heavy devices—which required extensive operational effort. The new solution offers sizes better aligned with our needs, allowing us to achieve optimal results with fewer logistical steps. Additionally, the latest generation is faster and provides more connectivity options at data centre premises, enhancing both efficiency and flexibility for large-scale data transfers.” - Lukasz Konarzewski, Senior Data Architect, Commvault “We have been using the devices to move 1PB of archival media data to Azure blob storage using the Data Box transfer devices. The next generation devices provided a very smooth setup and copy experience, and we were able to transfer data in larger chunks and much faster than before. Overall, this has helped shorten our migration lead times and land the data in the cloud quickly and seamlessly.” - Daniel Perry, Kohler “We have had a positive experience overall with the new Data Box devices to move our data to Azure Blob storage. The devices offer easy plug and play installation, detailed documentation especially for the security features and good data copy performance. We would definitely consider using it again for future large data migration projects.” – Bas Boeijink, Cloud Engineer, Eurofiber Cloud Infra Sign up for the Preview The Preview is available in the US, Canada, EU, UK, and US Gov Azure regions, and we will continue to expand to more regions in the coming months. If you are interested in the preview, we want to hear from you. Customers can sign up here ISV partners can sign up here You can learn more about the all-new Data Box devices here. We are committed to continuing to deliver innovative solutions to lower the barrier for bringing data to Azure. Your feedback is important to us. Tell us what you think about the new Azure Data Box preview by writing to us at DataBoxPM@microsoft.com – we can’t wait to hear from you. Stop by and see us! Now that you’ve heard about the latest innovation in the product family, do come by and see the new devices at the Ignite session What’s new in Azure Storage: Supercharge your data centric workloads, on 21st November starting 11:00 AM CST. You can also drop by the Infra Hub to learn more from our product experts and sign up to try the new devices for your next migration!Building a Scalable Web Crawling and Indexing Pipeline with Azure storage and AI Search
In the ever-evolving world of data management, keeping search indexes up-to-date with dynamic data can be challenging. Traditional approaches, such as manual or scheduled indexing, are resource-intensive, delay-prone, and difficult to scale. Azure Blob Trigger combined with an AI Search Indexer offers a cutting-edge solution to overcome these challenges, enabling real-time, scalable, and enriched data indexing. This blog explores how Blob Trigger, integrated with Azure Cognitive Search, transforms the indexing process by automating workflows and enriching data with AI capabilities. It highlights the step-by-step process of configuring Blob Storage, creating Azure Functions for triggers, and seamlessly connecting with an AI-powered search index. The approach leverages Azure's event-driven architecture, ensuring efficient and cost-effective data management.
Set Up Endpoint DLP Evidence Collection on your Azure Blob Storage
Endpoint Data Loss Prevention (Endpoint DLP) is part of the Microsoft Purview Data Loss Prevention (DLP) suite of features you can use to discover and protect sensitive items across Microsoft 365 services. Microsoft Endpoint DLP allows you to detect and protect sensitive content across onboarded Windows 10, Windows 11 and macOS devices. Learn more about all of Microsoft's DLP offerings. Before you start setting up the storage, you should review Get started with collecting files that match data loss prevention policies from devices | Microsoft Learn to understand the licensing, permissions, device onboarding and your requirements. Prerequisites Before you begin, ensure the following prerequisites are met: You have an active Azure subscription. You have the necessary permissions to create and configure resources in Azure. You have setup endpoint Data Loss Prevention policy on your devices Configure the Azure Blob Storage You can follow these steps to create an Azure Blob Storage using the Azure portal. For other methods refer to Create a storage account - Azure Storage | Microsoft Learn Sign in to the Azure Storage Accounts with your account credentials. Click on + Create On the Basics tab, provide the essential information for your storage account. After you complete the Basics tab, you can choose to further customize your new storage account, or you accept the default options and proceed. Learn more about azure storage account properties Once you have provided all the information click on the Networking tab. In network access, select Enable public access from all networks while creating the storage account. Click on Review + create to validate the settings. Once the validation passes, click on Create to create the storage Wait for deployment of the resource to be completed and then click on Go to resource. Once the newly created Blob Storage is opened, on the left panel click on Data Storage -> Containers Click on + Containers. Provide the name and other details and then click on Create Once your container is successfully created, click on it. Assign relevant permissions to the Azure Blob Storage Once the container is created, using Microsoft Entra authorization, you must configure two sets of permissions (role groups) on it: One for the administrators and investigators so they can view and manage evidence One for users who need to upload items to Azure from their devices Best practice is to enforce least privilege for all users, regardless of role. By enforcing least privilege, you ensure that user permissions are limited to only those permissions necessary for their role. We will use portal to create these custom roles. Learn more about custom roles in Azure RBAC Open the container and in the left panel click on Access Control (IAM) Click on the Roles tab. It will open a list of all available roles. Open context menu of Owner role using ellipsis button (…) and click on Clone. Now you can create a custom role. Click on Start from scratch. We have to create two new custom roles. Based on the role you are creating enter basic details like name and description and then click on JSON tab. JSON tab gives you the details of the custom role including the permissions added to that role. For owner role JSON looks like this: Now edit these permissions and replace them with permissions required based on the role: Investigator Role: Copy the permissions available at Permissions on Azure blob for administrators and investigators and paste it in the JSON section. User Role: Copy the permissions available at Permissions on Azure blob for usersand paste it in the JSON section. Once you have created these two new roles, we will assign these roles to relevant users. Click on Role Assignments tab, then on Add + and on Add role assignment. Search for the role and click on it. Then click on Members tab Click on + Select Members. Add the users or user groups you want to add for that role and click on Select Investigator role – Assign this role to users who are administrators and investigators so they can view and manage evidence User role – Assign this role to users who will be under the scope of the DLP policy and from whose devices items will be uploaded to the storage Once you have added the users click on Review+Assign to save the changes. Now we can add this storage to DLP policy. For more information on configuring the Azure Blob Storage access, refer to these articles: How to authorize access to blob data in the Azure portal Assign share-level permissions. Configure storage in your DLP policy Once you have configured the required permissions on the Azure Blob Storage, we will add the storage to DLP endpoint settings. Learn more about configuring DLP policy Open the storage you want to use. In left panel click on Data Storage -> Containers. Then select the container you want to add to DLP settings. Click on the Context Menu (… button) and then Container Properties. Copy the URL Open the Data Loss Prevention Settings. Click on Endpoint Settings and then on Setup evidence collection for file activities on devices. Select Customer Managed Storage option and then click on Add Storage Give the storage name and copy the container URL we copied. Then click on Save. Storage will be added to the list. Storage will be added to the list for use in the policy configuration. You can add up to 10 URLs Now open the DLP endpoint policy configuration for which you want to collect the evidence. Configure your policy using these settings: Make sure that Devices is selected in the location. In Incident reports, toggle Send an alert to admins when a rule match occurs to On. In Incident reports, select Collect original file as evidence for all selected file activities on Endpoint. Select the storage account you want to collect the evidence in for that rule using the dropdown menu. The dropdown menu shows the list of storages configured in the endpoint DLP settings. Select the activities for which you want to copy matched items to Azure storage Save the changes Please reach out to the support team if you face any issues. We hope this guide is helpful and we look forward to your feedback. Thank you, Microsoft Purview Data Loss Prevention TeamMigrate the critical file data you need to power your applications
When you migrate applications to Azure, you cannot leave file data behind. The Azure File Migration program can help you migrate data from NFS, SMB, and S3 sources to Azure Storage Services in less time, less risk, and no headache! Learn how to take advantage of this program and about the fundamentals of file migration in this post.Teach ChatGPT to Answer Questions: Using Azure AI Search & Azure OpenAI (Semantic Kernel)
In this two-part series, we will explore how to build intelligent service using Azure. In Series 1, we'll use Azure AI Search to extract keywords from unstructured data stored in Azure Blob Storage. In Series 2, we'll Create a feature to answer questions based on PDF documents using Azure OpenAITeach ChatGPT to Answer Questions: Using Azure AI Search & Azure OpenAI (Lang Chain)
In this two-part series, we will explore how to build intelligent service using Azure. In Series 1, we'll use Azure AI Search to extract keywords from unstructured data stored in Azure Blob Storage. In Series 2, we'll Create a feature to answer questions based on PDF documents using Azure OpenAI.
Unlocking Storage Optimizations: Smart Tiering for Blobs and ADLS in Azure Storage
We are excited to introduce the public preview of smart tier for Azure Blob and Azure Data Lake Storage. Smart tier is a fully managed, automated data tiering solution that takes the guesswork and manual effort out of optimizing your storage costs. Smart tier continuously analyzes your data’s access patterns and automatically moves objects between the hot, cool, and cold tiers. Smart tier will keep regularly accessed objects on the hot capacity tier to optimize transaction costs and moves inactive objects after 30 days to the cool tier capacity tier and after an additional 60 days of inactivity to the cold capacity tier. If you access an object in cool or cold tiers again, it’s instantly promoted back to the hot tier, restarting the cycle. This ensures your data is always in the most cost-effective tier with zero manual intervention, making it the ideal online tier for datasets with mixed or unknown access patterns. Getting started Using smart tier is quick and easy: Enabling smart tier is simple: Just select smart tier as the default access tier through the storage account configuration for any storage account with zonal redundancy. Smart tier is available in all zonal public cloud regions, supporting both flat and hierarchical namespaces. Billing is straightforward: You will pay the regular hot, cool, and cold capacity rates, with no extra charges for tier transitions, early deletion, or data retrieval. Even moving existing objects into smart tier does not incur tier change fees. There’s just a small monitoring fee for the orchestration. Smart tier is configured at the account level. It can be configured via API or the Azure portal as the default access tier setting for new and existing storage accounts. Existing objects following the default access tier setting from the account will be moved to smart tier automatically. Objects that are explicitly tiered, i.e. to the hot tier, will remain in the same account and will not be moved to other capacity tiers. Smart tier will always keep small objects that are below 128 KiB in size in the hot capacity tier for efficiency and those objects will not incur a monitoring charge. If objects below 128 KiB increase in size, the smart tiering patterns apply for those objects as well. The automatic down tiering of inactive data, paired with the billing model simplifications of Smart tier can lead to large cost savings over time. In the metrics view of the storage account you can see the distribution across the capacity tiers for smart tiered objects by both object count and capacity. This account shows smart tier in action, moving inactive objects to the cool and cold capacity tier, thereby drastically reducing the capacity charges without any manual intervention. "2 years ago, Qumulo partnered with Microsoft to deliver the first truly elastic, unlimited capacity, fully managed file system, Azure Native Qumulo, which was built on Azure Blob," said Brandon Whitelaw SVP of Product at Qumulo. "Qumulo shared feedback with Microsoft on our ideal solution for data tiering and Microsoft clearly delivered, meeting all expectations. With today's smart tier announcement, Qumulo will immediately enhance our offerings with these new capabilities, delivering greater functionality and control over data lifecycle management. We are thrilled with the feature set Azure is delivering at launch” Note that smart tier is not supported with append and page blobs. Smart tier is the ideal tier to choose when you are looking to store your data on standard online tiers but are not fully aware of the data access patterns or do not want to manage data transitions across online tiers. Objects managed by smart tier are not subject to lifecycle management policies, ensuring that automated tiering decisions are based solely on access patterns. Smart tier for block blobs is now available in public preview for both Azure Blob Storage and Azure Data Lake Storage for storage accounts with zonal redundancies, including ZRS, GZRS and RA-GZRS. Unlock cost savings by adding smart tier to your blob storage accounts in one easy step: https://aka.ms/BlobSmarttier. Please reach out to us for any feedback or questions, we would love to hear from you: smartblob@microsoft.com
