Understanding Azure storage redundancy offerings
In this post we'll go over major Azure infrastructure terminology aiming to get more clarity on how various high availability and disaster recovery offerings within Azure storage work and then segway to discussing the redundancy options in more detail. Read away!Announcing the Next generation Azure Data Box Devices
Microsoft Azure Data Box offline data transfer solution allows you to send petabytes of data into Azure Storage in a quick, inexpensive, and reliable manner. The secure data transfer is accelerated by hardware transfer devices that enable offline data ingestion to Azure. Our customers use the Data Box family to move petabytes-scale data into Azure for backup, archival, data analytics, media and entertainment, training, and different workload migrations etc. We continue to get requests about moving truly massive amounts of data in a secure, simple and quick manner. We’ve heard you and to address your needs, we’ve designed a new, enhanced product to meet your data transfer needs. About the latest innovation in Azure Data Box Family Today, we’re excited to announce the preview of Azure Data Box 120 and Azure Data Box 525, our next-generation compact, NVMe-based Data Box devices. The new offerings reflect insights gained from working with our customers over the years and understanding their evolving data transfer needs. These new devices incorporate several improvements to accelerate offline data transfers to Azure, including: Fast copy - Built with NVMe drives for high-speed transfers and improved reliability and support for faster network connections Easy to use - larger capacity offering (525 TB) in a compact form-factor for easy handling Resilient - Ruggedized devices built to withstand rough conditions during transport Secure - Enhanced physical, hardware and software security features Broader availability – Presence in more Azure regions, meeting local compliance standards and regulations What’s new? Improved Speed & Efficiency NVMe devices offer faster data transfer rates, with copy speeds up to 7 GBps via SMB Direct on RDMA (100-GbE) for medium to large files, a 10x improvement in data transfer speeds as compared to previous generation devices. High-speed transfers to Azure with data upload up to 5x faster for medium to large files, reducing the lead time for your data to become accessible in the Azure cloud. Improved networking with support for up to 100 GbE connections, as compared to 10 GbE on the older generation of devices. Two options with usable capacity of 120 TB and 525 TB in a compact form factor meeting OSHA requirements. Devices ship the next day air in most regions. Learn more about the performance improvements on Data Box 120 and Data Box 525. Enhanced Security The next generation devices come with several new physical, hardware and software security enhancements. This is in addition to the built in Azure security baseline for Data Box and Data Box service security measures currently supported by the service. Secure boot functionality with hardware root of trust and Trusted Platform Module (TPM) 2.0. Custom tamper-proof screws and built-in intrusion detection system to detect unauthorized device access. AES 256-bit BitLocker software encryption for data at rest is currently available. Hardware encryption via the RAID controller, which will be enabled by default on these devices, is coming soon. Furthermore, once available, customers can enable double encryption through both software and hardware encryption to meet their sensitive data transfer requirements. These ISTA 6A compliant devices are built to withstand rough conditions during shipment while keeping both the device and your data safe and intact. Learn more about the enhanced security features on Data Box 120 and Data Box 525. Broader Azure region coverage Recurring request from our customers has been for wider availability of our higher-capacity device to ease large migrations. We’re happy to share Data Box 525 will be available across most Azure regions where the Data Box service is currently live. This marks a significant improvement in availability of a large-capacity device as compared to the current Data Box Heavy. What our customers have to say For the last several months, we’ve been working directly with our customers of all industries and sizes to leverage the next generation devices for their data migration needs. Customers love the larger capacity with form-factor familiarity, seamless set up and faster copy. “This new offering brings significant advantages, particularly by simplifying our internal processes. With deployments ranging from hundreds of terabytes to even petabytes, we previously relied on multiple regular Data Box devices—or occasionally Data Box Heavy devices—which required extensive operational effort. The new solution offers sizes better aligned with our needs, allowing us to achieve optimal results with fewer logistical steps. Additionally, the latest generation is faster and provides more connectivity options at data centre premises, enhancing both efficiency and flexibility for large-scale data transfers.” - Lukasz Konarzewski, Senior Data Architect, Commvault “We have been using the devices to move 1PB of archival media data to Azure blob storage using the Data Box transfer devices. The next generation devices provided a very smooth setup and copy experience, and we were able to transfer data in larger chunks and much faster than before. Overall, this has helped shorten our migration lead times and land the data in the cloud quickly and seamlessly.” - Daniel Perry, Kohler “We have had a positive experience overall with the new Data Box devices to move our data to Azure Blob storage. The devices offer easy plug and play installation, detailed documentation especially for the security features and good data copy performance. We would definitely consider using it again for future large data migration projects.” – Bas Boeijink, Cloud Engineer, Eurofiber Cloud Infra Sign up for the Preview The Preview is available in the US, Canada, EU, UK, and US Gov Azure regions, and we will continue to expand to more regions in the coming months. If you are interested in the preview, we want to hear from you. Customers can sign up here ISV partners can sign up here You can learn more about the all-new Data Box devices here. We are committed to continuing to deliver innovative solutions to lower the barrier for bringing data to Azure. Your feedback is important to us. Tell us what you think about the new Azure Data Box preview by writing to us at DataBoxPM@microsoft.com – we can’t wait to hear from you. Stop by and see us! Now that you’ve heard about the latest innovation in the product family, do come by and see the new devices at the Ignite session What’s new in Azure Storage: Supercharge your data centric workloads, on 21st November starting 11:00 AM CST. You can also drop by the Infra Hub to learn more from our product experts and sign up to try the new devices for your next migration!Building a Scalable Web Crawling and Indexing Pipeline with Azure storage and AI Search
In the ever-evolving world of data management, keeping search indexes up-to-date with dynamic data can be challenging. Traditional approaches, such as manual or scheduled indexing, are resource-intensive, delay-prone, and difficult to scale. Azure Blob Trigger combined with an AI Search Indexer offers a cutting-edge solution to overcome these challenges, enabling real-time, scalable, and enriched data indexing. This blog explores how Blob Trigger, integrated with Azure Cognitive Search, transforms the indexing process by automating workflows and enriching data with AI capabilities. It highlights the step-by-step process of configuring Blob Storage, creating Azure Functions for triggers, and seamlessly connecting with an AI-powered search index. The approach leverages Azure's event-driven architecture, ensuring efficient and cost-effective data management.
Set Up Endpoint DLP Evidence Collection on your Azure Blob Storage
Endpoint Data Loss Prevention (Endpoint DLP) is part of the Microsoft Purview Data Loss Prevention (DLP) suite of features you can use to discover and protect sensitive items across Microsoft 365 services. Microsoft Endpoint DLP allows you to detect and protect sensitive content across onboarded Windows 10, Windows 11 and macOS devices. Learn more about all of Microsoft's DLP offerings. Before you start setting up the storage, you should review Get started with collecting files that match data loss prevention policies from devices | Microsoft Learn to understand the licensing, permissions, device onboarding and your requirements. Prerequisites Before you begin, ensure the following prerequisites are met: You have an active Azure subscription. You have the necessary permissions to create and configure resources in Azure. You have setup endpoint Data Loss Prevention policy on your devices Configure the Azure Blob Storage You can follow these steps to create an Azure Blob Storage using the Azure portal. For other methods refer to Create a storage account - Azure Storage | Microsoft Learn Sign in to the Azure Storage Accounts with your account credentials. Click on + Create On the Basics tab, provide the essential information for your storage account. After you complete the Basics tab, you can choose to further customize your new storage account, or you accept the default options and proceed. Learn more about azure storage account properties Once you have provided all the information click on the Networking tab. In network access, select Enable public access from all networks while creating the storage account. Click on Review + create to validate the settings. Once the validation passes, click on Create to create the storage Wait for deployment of the resource to be completed and then click on Go to resource. Once the newly created Blob Storage is opened, on the left panel click on Data Storage -> Containers Click on + Containers. Provide the name and other details and then click on Create Once your container is successfully created, click on it. Assign relevant permissions to the Azure Blob Storage Once the container is created, using Microsoft Entra authorization, you must configure two sets of permissions (role groups) on it: One for the administrators and investigators so they can view and manage evidence One for users who need to upload items to Azure from their devices Best practice is to enforce least privilege for all users, regardless of role. By enforcing least privilege, you ensure that user permissions are limited to only those permissions necessary for their role. We will use portal to create these custom roles. Learn more about custom roles in Azure RBAC Open the container and in the left panel click on Access Control (IAM) Click on the Roles tab. It will open a list of all available roles. Open context menu of Owner role using ellipsis button (…) and click on Clone. Now you can create a custom role. Click on Start from scratch. We have to create two new custom roles. Based on the role you are creating enter basic details like name and description and then click on JSON tab. JSON tab gives you the details of the custom role including the permissions added to that role. For owner role JSON looks like this: Now edit these permissions and replace them with permissions required based on the role: Investigator Role: Copy the permissions available at Permissions on Azure blob for administrators and investigators and paste it in the JSON section. User Role: Copy the permissions available at Permissions on Azure blob for usersand paste it in the JSON section. Once you have created these two new roles, we will assign these roles to relevant users. Click on Role Assignments tab, then on Add + and on Add role assignment. Search for the role and click on it. Then click on Members tab Click on + Select Members. Add the users or user groups you want to add for that role and click on Select Investigator role – Assign this role to users who are administrators and investigators so they can view and manage evidence User role – Assign this role to users who will be under the scope of the DLP policy and from whose devices items will be uploaded to the storage Once you have added the users click on Review+Assign to save the changes. Now we can add this storage to DLP policy. For more information on configuring the Azure Blob Storage access, refer to these articles: How to authorize access to blob data in the Azure portal Assign share-level permissions. Configure storage in your DLP policy Once you have configured the required permissions on the Azure Blob Storage, we will add the storage to DLP endpoint settings. Learn more about configuring DLP policy Open the storage you want to use. In left panel click on Data Storage -> Containers. Then select the container you want to add to DLP settings. Click on the Context Menu (… button) and then Container Properties. Copy the URL Open the Data Loss Prevention Settings. Click on Endpoint Settings and then on Setup evidence collection for file activities on devices. Select Customer Managed Storage option and then click on Add Storage Give the storage name and copy the container URL we copied. Then click on Save. Storage will be added to the list. Storage will be added to the list for use in the policy configuration. You can add up to 10 URLs Now open the DLP endpoint policy configuration for which you want to collect the evidence. Configure your policy using these settings: Make sure that Devices is selected in the location. In Incident reports, toggle Send an alert to admins when a rule match occurs to On. In Incident reports, select Collect original file as evidence for all selected file activities on Endpoint. Select the storage account you want to collect the evidence in for that rule using the dropdown menu. The dropdown menu shows the list of storages configured in the endpoint DLP settings. Select the activities for which you want to copy matched items to Azure storage Save the changes Please reach out to the support team if you face any issues. We hope this guide is helpful and we look forward to your feedback. Thank you, Microsoft Purview Data Loss Prevention TeamMigrate the critical file data you need to power your applications
When you migrate applications to Azure, you cannot leave file data behind. The Azure File Migration program can help you migrate data from NFS, SMB, and S3 sources to Azure Storage Services in less time, less risk, and no headache! Learn how to take advantage of this program and about the fundamentals of file migration in this post.Teach ChatGPT to Answer Questions: Using Azure AI Search & Azure OpenAI (Semantic Kernel)
In this two-part series, we will explore how to build intelligent service using Azure. In Series 1, we'll use Azure AI Search to extract keywords from unstructured data stored in Azure Blob Storage. In Series 2, we'll Create a feature to answer questions based on PDF documents using Azure OpenAITeach ChatGPT to Answer Questions: Using Azure AI Search & Azure OpenAI (Lang Chain)
In this two-part series, we will explore how to build intelligent service using Azure. In Series 1, we'll use Azure AI Search to extract keywords from unstructured data stored in Azure Blob Storage. In Series 2, we'll Create a feature to answer questions based on PDF documents using Azure OpenAI.Azure Blob Storage SFTP: General Availability of ACLs (Access Control Lists) of local users
We are excited to announce the general availability of ACLs (Access Control Lists) for Azure Blob Storage SFTP local users. ACLs make it simple and intuitive for administrators to manage fine-grained access control to blobs and directories for Azure Blob Storage SFTP local users. Azure Blob Storage SFTP Azure Blob storage supports the SSH File Transfer Protocol (SFTP) natively. SFTP on Azure Blob Storage lets you securely connect to and interact with the contents of your storage account by using an SFTP client, allowing you to use SFTP for file access, file transfer, and file management. Learn more here. Azure Blob Storage SFTP is used by a significant number of our customers, who have shared overwhelmingly positive feedback. It eliminates the need for third-party or custom SFTP solutions involving cumbersome maintenance steps such as VM orchestration. Local users Azure Blob Storage SFTP utilizes a new form of identity management called local users. Local users must use either a password or a Secure Shell (SSH) private key credential for authentication. You can have a maximum of 25,000 local users for a storage account. Learn more about local users here. Access Control for local users There are two ways in which access control can be attained for local users. 1. Container permissions By using container permissions, you can choose which containers you want to grant access to and what level of access you want to provide (Read, Write, List, Delete, Create, Modify Ownership, and Modify Permissions). Those permissions apply to all directories and subdirectories in the container. Learn more here. 2. ACLs for local users What are ACLs? ACLs (Access Control Lists) let you grant "fine-grained" access, such as write access to a specific directory or file, which isn’t possible with Container Permissions. More fine-grained access control has been a popular ask amongst our customers, and we are very excited to make this possible now with ACLs. A common ACL use case is to restrict a user's access to a specific directory without letting that user access other directories within the same container. This can be repeated for multiple users so that they each have granular access to their own directory. Without ACLs, this would require a container per local user. ACLs also make it easier for administrators to manage access for multiple local users with the help of groups. Learn more about ACLs for local users here. How to set and modify the ACL of a file or a directory? You can set and modify the permission level of the owning user, owning group, and all other users of an ACL by using an SFTP client. You can also change the owning user or owning group of a blob or directory. These operations require 'Modify Permissions' and 'Modify Ownership' container permissions, respectively. Note: Owning users can now also modify the owning group and permissions of a blob or directory without container permissions. This is a new feature enhancement added during the General Availability phase of ACLs for local users. For any user that is not the owning user, container permissions are still required. Learn more here. These enhancements significantly improve the management and usability of Azure Blob Storage SFTP by providing more granular access control over the container model and extending customer options. Please reach out to blobsftp@microsoft.com for feedback about SFTP for Azure Blob Storage. We look forward to your continued support as we strive to deliver the best possible solutions for your needs.
