FastTrack for Azure articles

Modernizing On‑Prem File Servers: Azure Storage Mover and File Sync

SriniThumala — Sun, 08 Mar 2026 21:46:02 GMT

Azure File Sync is a hybrid cloud storage service that centralizes on-premises file shares into Azure Files while preserving the experience of a local Windows file server. It installs an agent on Windows Server(s) to cache and sync files to an Azure file share (the cloud backend). Key features include cloud tiering (keeping only hot files on-prem and tiering cold data to Azure) and multi-site synchronization, so changes propagate across servers via the cloud. In essence, Azure File Sync transforms your file server into a cache for Azure, providing continuous two-way sync between on-premises and the Azure file share.

Azure Storage Mover is a fully managed migration service used to transfer file data into Azure Storage (Azure Blob containers or Azure file shares) with minimal downtime. It works by deploying a migration agent near the source storage, which then copies data directly to Azure. The cloud-based Storage Mover resource orchestrates migrations (including initial bulk transfer and optional delta syncs for changes) across multiple shares from a central interface. Unlike File Sync, Storage Mover is not a continuous sync service but rather a one-directional data mover for scenarios like one-time migrations or periodic updates.

Side-by-Side Technical Comparison

Aspect	Azure File Sync	Azure Storage Mover
Primary Purpose	*Hybrid file service*: Ongoing two-way sync between on-prem file servers and Azure Files, enabling local caching and multi-site data sharing.	*Migration tool*: One-way transfers of file data from on-prem (or other storage) to Azure Storage, optimized for lift-and-shift migrations with minimal downtime.
Architecture	Agent on Windows Server connects local NTFS volumes to an Azure File Share (cloud endpoint). The Azure Storage Sync Service coordinates sync across servers in a sync group. Supports cloud tiering to offload cold files to cloud. Sync is continuous and multi-directional (all endpoints stay in sync).	Cloud service + on-prem agent: An Azure Storage Mover resource manages migration jobs. Lightweight migration agents (VMs or containers) run near your sources, sending data directly to the Azure target (Blob or File share). The service orchestrates project-based migrations but does not keep sources in sync after completion.
Supported Sources	Windows file servers (NTFS), accessed via SMB/NFS protocols (the agent needs Windows Server OS). Ideal for Windows-based file shares.	SMB shares, NFS exports, and similar file systems on any platform (Windows or Linux NAS). Also supports migrating from other clouds’ storage (e.g., S3 buckets) into Azure. Broad support for heterogeneous sources.
Supported Targets	Azure Files only. (Cloud endpoint is an Azure file share in a Storage Account). Supports SMB (and NFS 4.1 Azure file shares in preview) as target share types.	Azure Storage (Blob containers or Azure file shares). For example, can migrate into an Azure Blob (ADLS Gen2) container or an Azure File share depending on scenario.
Sync vs. Migration	Continuous Sync: Bi-directional; changes on-prem or in Azure propagate to all endpoints. Designed for long-term hybrid operation, not just a one-time move.	Batch Migration: One-time or repeated transfer; not a live sync. Typically used to move data entirely to Azure (cutover once done). Supports incremental (delta) syncs to capture changes between migration runs.
Performance & Scale	Scales to large datasets (tested up to 100 million files per sync group). Throughput can reach hundreds of files/sec for upload/download given sufficient resources. Performance depends on server hardware, network, and Azure Files limits.	Built to handle high-volume migrations (100M+ files). Can scale out by deploying multiple agents or using bigger VMs to increase throughput. Performance mainly limited by network bandwidth and source/target IOPS and can be optimized by parallel jobs and delta sync workflows.
Integration	Deep integration with Azure Files (the back-end store) and Windows Server. Works with Azure Backup for centralized backups or using file share snapshots. Leverages Azure’s redundancy (LRS/ZRS/GRS) for durability; supports Azure AD DS for identity integration to maintain ACLs in cloud.	Integrated with Azure Arc (for agent management) and can combine with Azure Data Box for hybrid migrations (offline + online phases). Managed using Azure Portal and CLI, provides logging/monitoring through Azure Monitor for migration jobs. No ongoing infrastructure after migration completes.

Advantages of Azure File Sync:

Hybrid Cloud Caching: Retains on-premises low-latency file access (via local Windows servers) while using Azure as central storage. End users and apps continue using a local file server interface.
Cloud Tiering & Storage Efficiency: Frees up local storage by tiering infrequently used files to Azure. This reduces on-prem disk usage without sacrificing access to full dataset (files are pulled from cloud on demand).
Multi-site Sync & Collaboration: Enables near-real-time sync across multiple servers/sites via Azure hub. Great for distributed teams sharing a common file set, replacing need for complex DFS-R setups or manual transfers.
Minimal Disruption Migration: Can be used as a no-downtime migration path to Azure Files – sync in background, then cut over clients to the Azure share with identical structure and ACLs.

Advantages of Azure Storage Mover:

Purpose-Built for Migration: Optimized for transferring data at scale into Azure. Can handle large one-time migrations or scheduled recurrent syncs without continuous agent overhead post-migration. Simplifies multi-terabyte or multi-site migration projects.
Heterogeneous Source Support: Works with a variety of source types (SMB, NFS shares on any OS) and can migrate into both Azure Files and Azure Blob, providing flexibility that Azure File Sync can’t (e.g., migrating Linux NFS servers or third-party storage to Azure).
Centralized Orchestration: Cloud architects can manage all migrations via a single Azure Storage Mover resource – with projects & jobs tracking progress per share. Logging, error handling, and coordination are unified, unlike scripting copy operations per server.
Incremental & Low Downtime: Supports delta synchronization to bring the target up-to-date after an initial bulk copy. This reduces cutover downtime since only last-minute changes need transferring. Also integrates with offline seeding (Data Box) plus online catch-up to minimize network strain and downtime.

Common Use Cases and When to Choose Each

Azure File Sync – Use Cases: Ideal when you need to maintain on-premises file server access while leveraging cloud storage. Some common scenarios:

Branch Office File Sharing: Multiple offices each have a local file server, all synced to a central Azure Files share. Users get fast local access, and the cloud ensures each site’s data stays consistent.
File Server Augmentation: You want to extend an existing Windows file server with virtually unlimited cloud capacity (via tiering) rather than fully moving to cloud. Azure File Sync offloads old data and provides cloud backup, but users and apps continue as normal with the local server.
Gradual Migration / Testing: You plan to eventually migrate to Azure Files but want a seamless, no-downtime transition. Deploy AFS on the server, let it sync all data to cloud, then optionally eliminate the on-prem server later. This way, users never stop accessing their files during the migration process.

Azure Storage Mover – Use Cases: Best when you have a defined migration project to move file shares to Azure, especially for heterogeneous environments or large volumes:

Data Center Exit / Large File Share Migration: Moving tens or hundreds of TBs of data from on-prem NAS or file servers to Azure Storage as a one-time project. The Storage Mover’s robust scalability and delta-sync capabilities help ensure a smooth transfer with minimal final cutover downtime.
Consolidating Cross-Platform Data to Azure: If you need to migrate non-Windows file systems (Linux NFS, etc.) or even data from other clouds into Azure, Storage Mover supports those sources out-of-the-box. For example, migrating a Linux file repository into Azure Blob for big data analytics.
Recurring Scheduled Migrations: In cases where you periodically copy data from on-prem to Azure (e.g., monthly exports from a local system to cloud for archiving), Storage Mover can be run as needed and centrally monitored, without maintaining a constant sync infrastructure in between.

Conclusion

When to choose which: Use Azure File Sync if your goal is to keep using on-prem servers and need a hybrid solution for the foreseeable future, or if you require distributed caching and continuous sync for collaboration. It’s essentially part of your production architecture for hybrid cloud file storage. Conversely, choose Azure Storage Mover when you want to permanently migrate data into Azure (and possibly decommission on-prem storage), or when dealing with a one-off bulk transfer task. In summary, Azure File Sync is an ongoing hybrid file service, and Azure Storage Mover is a one-time (or scheduled) migration service. Both can complement your cloud strategy, but they address distinct scenarios in a cloud architect’s toolkit.

Azure Firewall and Service Endpoints

cloudtrooper — Mon, 14 Apr 2025 11:48:33 GMT

In my recent blog series Private Link reality bites I briefly mentioned the possibility of inspecting Service Endpoints with Azure Firewall, and many have asked for more details on that configuration. Here we go!

First things first: what the heck am I talking about? Most Azure services such as Azure Storage, Azure SQL and many others can be accessed directly over the public Internet. However, there are two alternatives to access those services over Microsoft's backbone: Private Link and VNet Service Endpoints. Microsoft's overall recommendation is using private link, but some organizations prefer leveraging service endpoints. Feel free to read this post on a comparison of the two.

You might want to inspect traffic to Azure services with network firewalls, even if that traffic is leveraging service endpoints. Before doing so, please consider that sending high-bandwidth traffic through a firewall might have cost implications and impact on the overall application latency. If you still want to go ahead, this post is going to explain how to do it.

The Design

Service endpoints have two configuration parts:

Source subnet configuration to tunnel traffic to the destination service.
Destination service configuration to accept traffic from the source subnet.

The key concept to understand is that if traffic from the client is going to be inspected by a firewall before going to the Azure service, then the source subnet is actually the Azure Firewall's subnet, not the original client's subnet:

You can configure service endpoints for a specific Azure service on a subnet using the portal, Terraform, Bicep, PowerShell or the Azure CLI. In the portal this is what it looks like:

For Azure CLI, enabling service endpoints for Azure Storage accounts in all regions would look like this:

❯ subnet_name=AzureFirewallSubnet
❯ az network vnet subnet update -n $subnet_name --vnet-name $vnet_name -g $rg --service-endpoints Microsoft.Storage.Global -o none --only-show-errors

You would then configure your Azure services to accept traffic coming from the Azure Firewall subnet. For example, for Azure Storage Accounts this is what you would see in the portal:

Network Rules or Application Rules?

Ideally you should use Application Rules in your firewall to make sure that your workloads are accessing the right Azure services, and not exfiltrating data to rogue data services that might be owned by somebody else and still could have the same IP address.

This is an example of a star rule granting access to all Azure Storage Accounts, but you should specify your own:

I tested with two storage accounts, one in the same region and another one in a different region than the client (the client being in this case the Azure Firewall). Access to both storage accounts is working, and as you can see both accesses are logged in the Storage Account as well as in the Azure Firewall (I have removed some characters from the storage account names to obfuscate them):

❯ ssh $vm_pip "curl -s4 $storage_blob_fqdn1"
Hello world! 
❯ ssh $vm_pip "curl -s4 $storage_blob_fqdn2"
Hello world!
❯ query='StorageBlobLogs | where TimeGenerated > ago(15m) | project AccountName, StatusCode, CallerIpAddress'
❯ az monitor log-analytics query -w $logws_customerid --analytics-query $query -o table
AccountName              CallerIpAddress     StatusCode  
----------------------   -----------------   ----------
storagetest????eastus2   10.13.76.72:10066   200
storagetest????westus2   10.13.76.72:11880   200
❯ query='AzureDiagnostics | where TimeGenerated > ago(15m) | where Category == "AZFWApplicationRule" | project SourceIP, Fqdn_s, Protocol_s, Action_s' 
❯ az monitor log-analytics query -w $logws_customerid --analytics-query $query -o table 
Action_s  Fqdn_s                                       Protocol_s SourceIP
--------- -------------------------------------------- ---------- ----------
Allow     storagetest????eastus2.blob.core.windows.net HTTPS      10.13.76.4 
Allow     storagetest????westus2.blob.core.windows.net HTTPS      10.13.76.4

The storage account sees as client IP the Azure Firewall's IP (in the subnet 10.13.76.64/26), and the Azure Firewall logs show the actual client IP (in the workload subnet 10.13.76.0/26).

If you use network rules you would lose a lot of the flexibility of the Azure Firewall, even if using FQDN-based rules. The reason is that if there were 2 storage accounts with different FQDNs sharing the same IP address, and the same client resolves both FQDNs, when Azure Firewall looks at the packet it will not be able to guess to which storage account each specific packet belongs to. From a routing perspective it would still work though, as long as you have the default SNAT settings of Azure Firewall, which involves translating the IP address when public IP addresses are involved:

Conclusion

There are some reasons why you might want to pick VNet service endpoints over private link (cost would probably be one of them). If so, there are advantages and disadvantages of sending traffic to Azure services via a firewall. If you decide to inspect traffic to VNet service endpoints with Azure Firewall, hopefully this post has shown you how to do that.

What are your thoughts about this?

Azure Private Endpoint vs. Service Endpoint: A Comprehensive Guide

SriniThumala — Fri, 31 Oct 2025 19:47:42 GMT

When building secure and scalable applications on Microsoft Azure, network connectivity becomes a critical factor. Azure provides two primary methods for enhancing security and connectivity: Private Endpoints and Service Endpoints. While both serve to establish secure connections to Azure resources, they function in distinct ways and cater to different networking needs. This blog will explain the differences between the two, their use cases, and when you should use each.

Understanding Service Endpoints

Azure Service Endpoints allow you to securely connect to Azure services over an optimized route through the Azure backbone network. When you enable service endpoints on a virtual network, they extend the private IP address space of that virtual network to the service. Essentially, they provide a direct, secure connection to Azure services like Azure Storage, Azure SQL Database, and Azure Key Vault without requiring the traffic to traverse the public internet.

Key Characteristics of Service Endpoints:

Public Services, Private IP: Service endpoints allow traffic to go through the Azure backbone but still access services using their public IP addresses. However, the traffic is not exposed to the internet.
Network Security Group (NSG) Integration: Service endpoints can be secured using NSGs, which control access based on source IP addresses and subnet configurations.
No DNS Resolution: Service endpoints use public DNS names to route traffic. Thus, the service endpoint enables network traffic to be routed privately but relies on public DNS resolution.

Use Cases for Service Endpoints:

Simplified Security: Service endpoints are ideal for connecting to Azure services in a straightforward manner without needing complex configurations.
Lower Latency: Since traffic is routed through the Azure backbone network, there’s less congestion compared to public internet traffic.
Integration with NSG: Service endpoints allow for tighter security control with Network Security Groups, ensuring only approved subnets and virtual networks can access specific services.

Understanding Private Endpoints

Private Endpoints, on the other hand, provide a direct, private connection to Azure resources by assigning a private IP address from your virtual network (VNet) to the service. Unlike service endpoints, which rely on public IPs, private endpoints fully encapsulate the service in a private address space. When a service is accessed via a private endpoint, the connection stays within the Azure network, preventing exposure to the public internet.

Key Characteristics of Private Endpoints:

Private IP Connectivity: Private endpoints map Azure resources to a private IP in your VNet, ensuring all traffic remains private and not exposed to the internet.
DNS Resolution: Private endpoints also require DNS configuration so that the private IP address can be resolved for the associated Azure service. Azure offers automatic DNS resolution for private endpoints, but custom DNS configurations can also be set.
End-to-End Security: Since the connection is over a private IP, it adds an additional layer of security by preventing any egress or ingress to public networks.

Use Cases for Private Endpoints:

Critical Security: Private endpoints are perfect for applications requiring high security, such as those handling sensitive data, financial transactions, or proprietary business logic.
Strict Regulatory Compliance: If you are dealing with highly regulated industries, private endpoints provide a way to ensure your data is not exposed to the public internet.
Network Isolation: Private endpoints are suited for scenarios where you want to fully isolate your Azure resources from the internet and only allow access from within your VNet.

Key Differences: Private Endpoint vs. Service Endpoint

Feature	Private Endpoint	Service Endpoint
Connection Type	Uses a private IP address from your VNet	Uses a public IP address but traffic stays within the region
DNS Resolution	Requires DNS configuration to resolve private IPs	Relies on public DNS for resolution
Use Case	Ideal for critical security and isolated traffic	Best for connecting to Azure services with basic security requirements
Supported Services	Limited to resources that support private endpoints	Supports a broader range of Azure services like Storage, SQL, etc.

Note: Service endpoints are as secured as the private endpoints when configured appropriately.

When to Use Each Option

Choose Service Endpoints if:

You want to connect to Azure services like Storage, SQL, or Key Vault using the Azure backbone network.
Your security requirements do not mandate complete isolation from the public internet.
You need to leverage Network Security Groups (NSGs) to limit access from specific subnets or VNets.

Choose Private Endpoints if:

Your application requires full isolation from the public internet, such as for sensitive workloads or highly regulated data.
You want traffic to flow entirely within the private network, ensuring complete confidentiality.
You need to maintain strict security standards for applications that interact with services like databases, storage accounts, or other critical infrastructure.

Conclusion

Both Private Endpoints and Service Endpoints play vital roles in securing connectivity to Azure services, but they cater to different security needs. Service Endpoints offer an easier, simpler way to secure access over the Azure backbone, while Private Endpoints provide complete isolation and enhanced security by assigning a private IP address.

By carefully assessing your application's security needs and performance requirements, you can choose the appropriate method to ensure optimal connectivity and compliance with Azure services.

FastTrack for Azure (FTA) program retiring December 2024

JillArmourMicrosoft — Fri, 13 Dec 2024 20:21:12 GMT

ATTENTION:

As of December 31st, 2024, the FastTrack for Azure (FTA) program will be retired. FTA will support any projects currently in motion to ensure successful completion by December 31st, 2024, but will no longer accept new nominations.

For more information on available programs and resources, visit: Azure Migrate, Modernize, and Innovate | Microsoft Azure

Azure Backup vs. Azure Site Recovery: Key Differences Explained

SriniThumala — Tue, 10 Dec 2024 20:16:17 GMT

When it comes to safeguarding your data and ensuring business continuity, Microsoft Azure offers two powerful solutions: Azure Backup and Azure Site Recovery (ASR). Although both services are critical components of a comprehensive disaster recovery strategy, they serve distinct purposes. Based on my experience working with the hundreds of Customers sometimes they are not sure to use which services or their use cases. In some cases, Customers need both services to meet their business requirements.

Here's a breakdown of their key differences:

Purpose and Functionality

Azure Backup: This service focuses on data backup and restoration. It provides a simple, secure, and reliable way to back up files, folders, applications, and virtual machines (VMs) to Azure. Azure Backup protects against data loss due to accidental deletion, ransomware attacks, or corruption.
Azure Site Recovery (ASR): ASR is designed for disaster recovery and business continuity. It replicates workloads running on physical or virtual machines to a secondary location to ensure seamless failover during a disaster.

Use Case: Azure Backup is ideal for long-term data retention, whereas ASR is critical for minimizing downtime and ensuring workload availability during outages.

Core Capabilities

- Azure Backup:
  - Creates backups for Azure VMs, On-prem VMs, Azure Managed Disks, Azure file shares, SQL server in Azure VMs, SAP HANA databases in Azure VMs, Azure Blobs, Azure Kubernetes services and Azure Database for PostgreSQL servers.
  - Supports both on-premises and cloud-based resources.
  - Provides long-term retention and lifecycle management for backups.
  - Offers encryption at rest and in transit to secure data.
- Azure Site Recovery:
  - Replicate Azure VMs, On-premises VMs and VMWare VMs.
  - Continuous replication of workloads for low recovery point objectives (RPOs).
  - Orchestrated failover and failback capabilities.
  - Multi-region disaster recovery for VMs and physical servers.
  - Integration with BCDR (Business Continuity and Disaster Recovery) plans.

Key Differentiator: Azure Backup is about data recovery, while ASR is about workload continuity.

Recovery Objectives

- Azure Backup: Focuses on the Recovery Time Objective (RTO) and Recovery Point Objective (RPO) for restoring individual files or entire systems. RPO depends on the backup schedule (e.g., daily or hourly backups).
- Azure Site Recovery: Aims to minimize downtime by ensuring applications and workloads are quickly available in a secondary location during an outage. It delivers lower RTO and RPO compared to backup solutions.

Data Recovery vs. Workload Recovery

- Azure Backup: Restores data at a granular level (e.g., files, folders, or entire systems).
- Azure Site Recovery: Ensures entire workloads, including infrastructure and applications, are replicated and can be failed over to another location.

Cost

- Azure Backup: Costs are primarily based on the size of backed-up data and the number of recovery points stored in the Recovery Services Vault.
- Azure Site Recovery: Pricing is driven by the number of instances being replicated and the storage consumed by replicated data.

Comparison: Azure Backup is generally more cost-effective for data protection, whereas ASR justifies its higher cost by providing enterprise-grade disaster recovery capabilities.

Final Thoughts

Azure Backup and Azure Site Recovery are complementary solutions that address different aspects of data protection and disaster recovery. For long-term data retention and restoration, Azure Backup is the go-to solution. For mission-critical applications requiring business continuity during disruptions, Azure Site Recovery ensures workloads remain operational with minimal downtime.

A robust IT strategy often involves leveraging both services to cover the spectrum of data protection and recovery needs, ensuring business resilience no matter the scenario.

Oracle Database@Azure DNS Setup

adelagarde — Tue, 19 Nov 2024 15:47:53 GMT

Introduction

The following article describes considerations and recommendations for setting up Domain Name Service (DNS) when deploying Oracle Database@Azure. The goal of this article is to provide recommended technical guidance to enhance customer experience for a stable cloud environment. The article assumes that the reader has a basic understanding of Oracle Database technologies and Azure Compute and networking. As parts of your preparation and architecture design process refer to the following article for more information, see DNS for on-premises and Azure - Cloud Adoption Framework

Scenario

You want to migrate your on-premises Oracle databases to Oracle Database@Azure. During the migration, you need to ensure proper name resolution both on-premises and in Azure. Oracle Data Guard will be used for the data migration. After migration, you aim to maintain reliable name resolution using your hybrid connection without a custom DNS zone with applications that Oracle Database@Azure will communicate with.

DNS Deployment options for Oracle Database@Azure

The option that can be set up to satisfy the mentioned scenario and requirements would be to extend DNS infrastructure from on-premises into Azure with an IaaS (Infrastructure as a Service) solution.

DNS prerequisites before you deploy Oracle Database@Azure in Azure

Review the different DNS deployment scenario workflows with Oracle Database@Azure

Oracle Database@Azure default DNS setup workflow

Oracle Database@Azure custom DNS setup workflow

Oracle Database@Azure external DNS setup workflow

If you don't select to use a custom DNS domain, the Oracle Exadata uses the default domain, oraclevcn.com. The oraclvcn.com zone is auto provisioned in Azure as a private DNS zone linked to the Oracle Database@Azure virtual network. After Oracle Database@Azure is deployed, records from OCI begin to auto populate the oraclevcn.com private DNS zone in Azure.
Private view and private zone must be created before deploying the Exadata Cluster if you plan to use a custom DNS zone such as contoso.com with Oracle Database@Azure. Read this article for details Private DNS Resolvers. The private resolver created in the OCI Private view is not authoritative. It is only applied for Oracle Database@Azure for name resolution external to the OCI VCN compartment.
Forwarding to another DNS server should be set up beforehand in the OCI DNS console if your deployment scenario requires this. For details, see DNS in your Virtual Cloud Network.
Private zone's name can't have more than four labels. For example, a.b.c.d (oradb.prod.contoso.com) is allowed while a.b.c.d.e (oradb.sales.prod.contoso.com) isn't. This constraint is only when using a custom DNS zone such as contoso.com to provision the database cluster. Review the following Oracle article DNS in your Virtual Cloud Network.
When provisioning an Exadata VM Cluster using Private DNS feature, Exadata needs to create reverse DNS zones in the compartment of Exadata virtual machine (VM) Cluster. If the compartment defines tags or tag defaults, other policies related to managing tags are needed. For details, see:
Required Permissions for Working with Defined Tags
Required Permissions for Working with Tag Defaults

Custom DNS

Let us address the scenario, which is an Oracle database migration from on-premises to Azure. There are various tools and methods to migrate the Oracle database such as Oracle Data Guard as an example. Configuration details of Oracle Data Guard are outside the scope of this article.

Before configuring Oracle Data Guard for data replication from an on-premises database to Azure, it’s essential to ensure the on-premises database can resolve the FQDN of the Oracle Database@Azure instance. The Oracle Database@Azure cluster’s client network advertises three IPs, which are linked to a Single Client Access Name (SCAN) address. This SCAN address provides service access to clients connecting to Oracle Database@Azure and offers redundancy for the cluster. In the event of a node failure within the cluster, the SCAN FQDN maintains uninterrupted client connections during the failover process as the new primary node becomes active. This setup negates the necessity for manual DNS round-robin configurations, which were common in traditional on-premises Oracle database deployments.

Proper resolution of the SCAN FQDN is a prerequisite for establishing direct connectivity from the on-premises database to the Oracle Database@Azure instance in Azure. The name resolution from Oracle Database@Azure in Azure to resources in the datacenter would have to function. This connection could be facilitated through Azure ExpressRoute or a high-bandwidth VPN. Additionally, it’s crucial to confirm that Oracle Database@Azure can resolve the on-premises Oracle database using the custom DNS servers deployed in Azure. Once name resolution and network connectivity are assured, Oracle Data Guard can be configured to initiate data migration using the Oracle Database@Azure SCAN FQDN on TCP (Transmission Control Protocol) port 1521. The data migration can begin and once completed the eventual cutover to Azure from the legacy Oracle database.

The following diagram represents the implementation of a DNS solution with VMs (Virtual Machines) or appliances from the Azure marketplace with the network path that DNS name resolution would follow across your Azure Landing Zone (ALZ).

The VMs can be either Windows, Linux, FreeBSD, and active directory domain controllers with integrated DNS for name resolution. These systems in Azure would be integrated into the overall DNS infrastructure of your organization. The SCAN FQDN, on-premises databases, and on-premises legacy applications would be registered on DNS servers deployed across the enterprise.

Once the migration is complete applications like NFS shares, web servers, or an AKS (Azure Kubernetes Service) cluster in either Azure or the on-premises datacenter receives a client request initiating a DNS query. The DNS server returns the response of the SCAN FQDN associated with the Oracle Database@Azure cluster back to the application that initiated the query. This facilitates the connection to the database for either reading or writing operations.

You can query the SCAN FQDN from the oraclevcn.com DNS zone or apply your domain and naming convention in your DNS servers as a CNAME record. An example on how to create a CNAME in a Windows DNS server is in this article To add an alias (CNAME) resource record in a DNS zone.

If the application resides on Azure, it queries the DNS servers deployed in the custom DNS servers defined in the Azure virtual network. The on-premises applications would use the on-premises DNS servers defined in the datacenter.

Review the following recommendations to ensure this implementation supports both reliability and redundancy.

A DNS solution on IaaS deployed in Azure must have the Azure Wire Server 168.63.129.16 set as a conditional forwarder for proper name resolution with Azure Private DNS zones. Virtual network links needs to connect the Azure Private DNS zone with the virtual network that has the IaaS DNS resources. This includes zones such as oraclevcn.com and other private zones supporting other Azure services. Details about the Azure Wire Server are in this article What is IP 168.63.129.16 What is IP 168.63.129.16.
Virtual network links need to be created for each Azure Private DNS zone and connected to the virtual network where the DNS server VMs or the DNS appliance resides for proper name resolution to work.
Deploy at a minimum of two VMs in the same virtual network and same Azure region. Use a VMSS (Virtual Machine Scale Sets) Flex for redundancy and configure DNS settings on virtual networks to use the IPs of the custom DNS servers. Read recommendations for redundancy in this article Recommendations for designing redundancy

Please look for updates to this article and design patterns on Microsoft Learn and Microsoft Architecture Center.

Next Steps

Onboard Oracle Database@Azure

Network Planning for Oracle Database@Azure

Groups and Roles for Oracle Database@Azure

Overview of Provisioning

Connecting to Azure SQL Database using SQLAlchemy and Microsoft Entra authentication

franklinlindemberg — Thu, 31 Oct 2024 16:02:44 GMT

In this blog, we will focus on a common solution that demonstrates how to securely connect to an Azure SQL Database using Microsoft Entra Authentication with the current logged in user. It leverages the SQLAlchemy library for Python, integrating Entra's secure identity framework with your database connection.

Key Steps:

Set Current User as Admin: You begin by configuring an Azure Entra account as the admin for the Azure SQL Server.
Configure Firewall Rules: Ensure that your machine or application has access by adding its IP address to the Azure SQL Server firewall.
Create Secure Connection: Finally, the Python SQLAlchemy library is used to connect to the database, relying on Microsoft Entra authentication instead of hard-coded credentials.

With this setup, we achieve a secure, credential-less connection to Azure SQL Database!

Comparing Azure SQL Authentication Methods

Before diving into the solution, let's compare authentication methods. When it comes to securing access to your Azure SQL Database, the method you choose for authentication can significantly impact both the security and manageability of your applications. There are two primary methods commonly used: SQL Authentication, which relies on username and password credentials, and Microsoft Entra Managed Identity, which utilizes Microsoft Entra ID (formally Azure AD) for identity and access management.

SQL Authentication Drawbacks

SQL Authentication, while straightforward, comes with inherent security risks and management burdens. One of the main concerns is the reliance on hard-coded or stored credentials, often passed through connection strings in application code or configuration files. Additionally, using the stored static credentials allows continued access until explicitly revoked, enlarging your database's attack surface. For example, when using SQL Authentication, developers might include connection credentials like this:

connection_string = "Driver={SQL Server};Server=tcp:yourserver.database.windows.net,1433;Database=yourdb;Uid=yourusername;Pwd=yourpassword;"

In this example, embedding the username and password in the application introduces several vulnerabilities:

Credential Exposure: If the codebase is shared, leaked, or compromised, database credentials can be exposed.
Secret Management: You need solutions like Azure Key Vault to securely store and rotate credentials, adding complexity.
Credential Rotations: SQL credentials require manual or automated rotation, increasing operational overhead.

Improved Security with Microsoft Entra authentication

Microsoft Entra authentication (formerly known as Azure AD) offers a more secure and manageable way to authenticate applications and users to Azure SQL Database. Instead of relying on stored credentials, Microsoft Entra uses tokens generated dynamically and securely by Azure's identity management system, eliminating the need for static credentials in your applications or configuration files.

Key Security Advantages:

Credential-less Access: No need to store or transmit sensitive credentials (username and password) in code or configuration files.
Time-Limited Access: Entra-generated tokens have limited lifetimes, reducing the risk of misuse or unauthorized access over extended periods.
Centralized Management: Entra integrates seamlessly with other Azure services, providing centralized identity and access control across your applications.
Role-Based Access Control (RBAC): By using Entra authentication, access can be more finely tuned using RBAC, meaning users only get the permissions they need to perform their tasks.

In contrast to SQL Authentication, which requires manually revoking credentials, Microsoft Entra authentication ensures that when access to an account is revoked, it immediately affects all Azure services, preventing further unauthorized access. This vastly reduces the risk of security breaches due to stale credentials lingering in code repositories or configuration files.

Pre-requisites

An Azure subscription.
An Azure SQL database configured with Microsoft Entra authentication. You can create one using the Create database quickstart.
The latest version of the Azure CLI.
Visual Studio Code with the Python extension.
Python 3.8 or later.
ODBC Driver for SQL Server

Configure the Database

Setting Current User as Azure SQL DB Admin

First, you need to set your current Azure AD user as the Azure SQL Admin for your database. Follow the steps below:

Navigate to Your Azure SQL Server:
- Log in to the Azure Portal.
- Search for and select your Azure SQL Server (not the individual database).
Set Azure AD Admin:
- In the left-hand menu, under Settings, click on Microsoft Entra ID.
- Select Support Only Microsoft Entra authentication for this server to ensures no one can access the database server using SQL login credentials.
- Click on Set admin.
  - In the Add admin pane, search for your user account.
  - Select your account and click Select.
  - This will set your user as a database admin and allow it to login using Microsoft Entra authentication.
- Click on Save.

Adding Your IP Address to the Azure SQL Server Firewall

To ensure your connection to Azure SQL Database is secure and allowed, you will need to add your IP address to the server's firewall rules. This step prevents unauthorized IPs from accessing your server while allowing your trusted IP to connect. Follow these steps:

Navigate to Your Azure SQL Server:
- Log in to the Azure Portal.
- Search for and select your Azure SQL Server.
Configure Firewall Settings:
- In the left-hand menu under Security, select Networking.
- In the Public network access section, enable Selected networks to allow the firewall rule in order to whitelist your IP address.
- Under the Firewall rules section, click on Add your client IPv4 address. This will automatically detect your current IP address and add it to the list of allowed addresses.
- Click on Allow Azure services and resources to access this server. This will allow your web app running on Azure to access the database.
- Click on Save.

At this point, we have set up an Azure AD user as the admin for the Azure SQL Server, enforcing Entra ID (formerly Azure AD) authentication and eliminating the need for SQL login credentials. This reduces the risk of credential exposure while streamlining identity management. We also added your IP to the Azure SQL Server firewall whitelist, ensuring only authorized IP addresses can connect, minimizing exposure to external threats.

With these security measures in place, we are ready to securely connect and interact with the Azure SQL Database using Python, leveraging Microsoft for seamless, credential-free authentication.

Set up the project

Now that the database setup is complete, we are ready to implement and use the code that will interact with the database. We will be using SQLAlchemy, which provides many database capabilities for python developers, like ORM capabilities and connection pooling.

1. Open Visual Studio Code and create a new folder for your project and change directory into it.

mkdir python-sql-azure cd python-sql-azure

2. Create a requirements.txt file with the following content:

pyodbc fastapi uvicorn[standard] pydantic azure-identity sqlalchemy

3. Create a start.sh file (this is only needed if you plan to deploy this project to azure)

gunicorn -w 4 -k uvicorn.workers.UvicornWorker app:app

4. Create an app.py file with the content below:

import struct import urllib from typing import Union, Optional from fastapi import FastAPI, HTTPException from pydantic import BaseModel import sqlalchemy as db from sqlalchemy import String, select, event from sqlalchemy.orm import Session, Mapped, mapped_column from sqlalchemy.ext.declarative import declarative_base from azure.identity import DefaultAzureCredential driver_name = '{ODBC Driver 18 for SQL Server}' server_name = 'sql-fg-database-s4ujd' database_name = 'sqldb-fg-database-s4ujd' connection_string = 'Driver={};Server=tcp:{}.database.windows.net,1433;Database={};Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30'.format(driver_name, server_name, database_name) Base = declarative_base() credential = DefaultAzureCredential() class PersonSchema(BaseModel): first_name: str last_name: Union[str, None] = None class Person(Base): __tablename__ = "Persons" id: Mapped[int] = mapped_column(primary_key=True, name="ID") first_name: Mapped[str] = mapped_column(String(30), name="FirstName") last_name: Mapped[Optional[str]] = mapped_column(name="LastName") def __repr__(self) -> str: return f"Person(ID={self.id!r}, FirstName={self.first_name!r}, LastName={self.last_name!r})" def get_engine(): params = urllib.parse.quote(connection_string) url = "mssql+pyodbc:///?odbc_connect={0}".format(params) return db.create_engine(url) engine = get_engine() # from https://docs.sqlalchemy.org/en/20/core/engines.html#generating-dynamic-authentication-tokens @event.listens_for(engine, "do_connect") def provide_token(dialect, conn_rec, cargs, cparams): """ Called before the engine creates a new connection. Injects an EntraID token into the connection parameters. """ print('creating new token') token_bytes = credential.get_token("https://database.windows.net/.default").token.encode("UTF-16-LE") token_struct = struct.pack(f'<I{len(token_bytes)}s', len(token_bytes), token_bytes) SQL_COPT_SS_ACCESS_TOKEN = 1256 # This connection option is defined by microsoft in msodbcsql.h cparams["attrs_before"] = {SQL_COPT_SS_ACCESS_TOKEN: token_struct} # set up the database Base.metadata.create_all(engine) app = FastAPI() @app.get("/all") def get_persons(): with Session(engine) as session: stmt = select(Person) rows = [] for person in session.scalars(stmt): print(person.id, person.first_name, person.last_name) rows.append(f"{person.id}, {person.first_name}, {person.last_name}") return rows @app.get("/person/{person_id}") def get_person(person_id: int): with Session(engine) as session: stmt = select(Person).where(Person.id == person_id) person = session.execute(stmt).scalar() if not person: raise HTTPException(status_code=404, detail="Person not found") return f"{person.id}, {person.first_name}, {person.last_name}" @app.post("/person") def create_person(item: PersonSchema): with Session(engine) as session: person = Person(first_name=item.first_name, last_name=item.last_name) session.add(person) session.commit() return item

Notes:

make sure to update the server_name and database_name variables in the code above with the names you used to create both the SQL server and the database
The provide_token method will be called every time a database connection is created by the engine. It's responsible for injecting the EntraID token so it can successfully authenticate to the database. This is necessary in order to always have a fresh token when creating a connection, otherwise if we had a static token that was already expired, it would never be able to connect again to the database.

Running Locally

1. Create a virtual environment for the app

py -m venv .venv .venv\scripts\activate

2. Install requirements

pip install -r requirements.txt

3. Run the app.py file in Visual Studio Code.

uvicorn app:app --reload

4. Open the Swagger UI at http://127.0.0.1:8000/docs

5. Create a new user using the Create Person endpoint

6. Try Get Person and Get Persons endpoints

Running on Azure

1. Use the az webapp up to deploy the code to App Service.

az webapp up --resource-group <resource-group-name> --name <web-app-name>

2. Use the az webapp config set command to configure App Service to use the start.sh file.

az webapp config set --resource-group <resource-group-name> --name <web-app-name> --startup-file start.sh

3. Use the az webapp identity assign command to enable a system-assigned managed identity for the App Service. This is needed because we will grant database access to this identity, with specific roles.

az webapp identity assign --resource-group <resource-group-name> --name <web-app-name>

4. Grant permissions to the web app identity by running the SQL commands below on your database. The first commanda creates a database user for the web app and the following ones sets data reader/writer roles (you can find more details about roles at Database-level roles - SQL Server | Microsoft Learn). By doing this we guarantee that the web app has the least privilege.

CREATE USER [<web-app-name>] FROM EXTERNAL PROVIDER ALTER ROLE db_datareader ADD MEMBER [<web-app-name>] ALTER ROLE db_datawriter ADD MEMBER [<web-app-name>]

5. Open the Swagger UI at https://<web-app-name>.azurewebsites.net/docs and test the endpoints again

References

Connect to and query Azure SQL Database using Python and the pyodbc library - Azure SQL Database | Microsoft Learn

Securing Your Data Pipelines: Best Practices for Fabric Data Factory

Sally_Dabbah — Tue, 15 Oct 2024 08:16:49 GMT

Introduction

In today’s data-driven world, securing data pipelines is crucial to protect sensitive information and ensure compliance with regulatory requirements. Microsoft Fabric Data Factory experience (FDF) offers a robust set of security features to safeguard your data as it moves through various stages of your data workflows. In this post, we’ll explore key security features in FDF and demonstrate how to implement them with a practical example.

Key Security Features in Fabric Data Factory

Before diving into the implementation, let’s take a look at the primary security mechanisms that Fabric Data Factory provides:

Authentication: Fabric data factory uses on Microsoft Entra ID to authenticate users (or service principals). When authenticated, users receive access tokens from Microsoft Entra ID. Fabric uses these tokens to perform operations in the context of the user.
Authorization: All Fabric permissions are stored centrally by the metadata platform. Fabric services query the metadata platform on demand in order to retrieve authorization information and to authorize and validate user requests.
Data Encryption:
Data at rest: All Fabric data stores are encrypted at rest by using Microsoft-managed keys. Fabric data includes customer data as well as system data and metadata.
Data at transit: Data in transit between Microsoft services is always encrypted with at least TLS 1.2. Fabric negotiates to TLS 1.3 whenever possible. Traffic between Microsoft services always routes over the Microsoft global network.
Managed Identities: A Fabric workspace identity is an automatically managed service principal that can be associated with a Fabric workspace. Fabric workspaces with a workspace identity can securely read or write to firewall-enabled Azure Data Lake Storage Gen2 accounts through trusted workspace access for OneLake shortcuts. Fabric items can use the identity when connecting to resources that support Microsoft Entra authentication. Fabric uses workspace identities to obtain Microsoft Entra tokens without the customer having to manage any credentials.
Key Vault Integration: unfortunately, as of today Key vault integration is not supported in Dataflow Gen 2 / data pipeline connections in Fabric.
Network Security: When you connect to Pipeline via private link, you can use the data pipeline to load data from any data source with public endpoints into a private-link-enabled Microsoft Fabric Lakehouse. Customers can also author and operationalize data pipelines with activities, including Notebook and Dataflow activities, using the private link. However, copying data from and into a Data Warehouse isn't currently possible when Fabric's private link is enabled.

Now, let’s walk through an example that demonstrates how to secure data pipeline in Fabric Data Factory (FDF).

Example: Securing a Data Pipeline in Fabric Data Factory

Scenario:

You are setting up a data pipeline that moves sensitive data from ADLS gen 2 to Fabric warehouse. To ensure that this pipeline is secure, you will:

Use a managed identity to authenticate Fabric data factory
Configure trusted workspace access in ADLS Gen2
Copy data from ADLS to Fabric Lakehouse using secured pipeline

Prerequisites:
- Tools and Technologies Needed:

Azure Data Lake Gen2 (ADLS) storage account.
knowledge in Azure data factory.
Fabric workspace.

Steps:

Step 1: Enable Managed Identity in Workspace level for Fabric Data Factory pipeline

Workspace identity can be created and deleted by workspace admins. The workspace identity has the workspace contributor role on the workspace.

Workspace identity is supported for authentication to target resources in connections. Only users with an admin, member, or contributor role in the workspace can configure the workspace identity for authentication in connections.

Managed identities allow Fabric Data Factory to securely authenticate to other Azure services without hardcoding credentials.

Navigate to the workspace and open the workspace settings.
Select the Workspace identity tab.
Select the + Workspace identity button.

Once enabled, this identity can be used to access resources like Azure SQL Database securely.

Step 2: Configure trusted workspace access in ADLS Gen2

Sign in to the Azure portal and go to Custom deployment.
Choose Build your own template in the editor. For a sample ARM template that creates a resource instance rule, see ARM template sample.
Create the resource instance rule in the editor. When done, choose Review + Create.
On the Basics tab that appears, specify the required project and instance details. When done, choose Review + Create.
On the Review + Create tab that appears, review the summary and then select Create. The rule will be submitted for deployment.

When deployment is complete, you'll be able to go to the resource.

Step 3: Create a pipeline to connect to ADLS gen2 and copy data to Fabric Lakehouse

with this pipeline, we will connect directly to a firewall-enabled ADLS Gen2 account that has trusted workspace access enabled.

Navigate to your workspace, click on new item
Create new pipeline
In your pipeline, add Copy activity to the canvas
In Copy activity Source Tab : Choose ADLS Gen2 as data source and connect to it
In Destination tab, Connect to the lakehouse and select table

Step 4: Results

After copy activity finish running, you can view the data in your Lakehouse

Conclusion

- Securing your data pipelines in Azure Data Factory is essential to maintaining the integrity, confidentiality, and availability of your data. By leveraging features such as managed identities you can build a robust security framework around your data flows.

Do you have any other tips for securing Fabric Data Factory? Let me know in the comments!

- Follow me on LinkedIn: Sally Dabbah | LinkedIn

Making Searching and Curating Data Assets in Microsoft Purview easier.

Eduardo_Noriega — Mon, 30 Sep 2024 04:32:33 GMT

1. Introduction.

Currently, IT infrastructure stores and maintains data assets, even though IT doesn't own or use the data.

There's a disconnect between how data needs to be discovered and maintained within the business, and the teams that maintain it.

Without standardized procedures for data governance, data handling often relies on manual processes, leading to inefficiencies, data loss, insufficient data protection and higher operational costs.

Microsoft Purview is designed to help enterprises get the most value from their existing information assets.

The catalog makes data sources easily discoverable and understandable by the users who manage the data.

With Purview, organizations can gain insights into data lineage, data usage, and data connections, helping them to comply with regulations such as GDPR, CCPA, and others.

Microsoft Purview provides a cloud-based service into which you can register data sources. During registration, the data remains in its existing location, but a copy of its metadata is added to Microsoft Purview, along with a reference to the data source location.

After you register a data source, you can scan it and enrich its metadata.

Discovering and understanding data sources and their use is the primary purpose of registering the sources.

In this article, we describe smart features that allow you to search previously scanned data assets using natural language queries, along with automated metadata enrichment for curating these assets.

2. Data Catalog.

The Data Catalog is a Data Governance solution that enables business experts and technical data owners to collaborate and contribute to a shared understanding of data.

Among other functionalities, in Data Catalog you can use data search to discover data assets from multiple data sources, including Microsoft Fabric items and workspaces Exploring the Relationship Between Microsoft Fabric and Microsoft Purview: What You Need to Know

Microsoft Purview’s Smart Data Searching primarily works with scanned data assets. For unscanned data assets, manual classification and tagging can be done, but they may not fully leverage the capabilities of Smart Data Searching. For real-time or “Live View” data, you would typically need to scan the data source first to make it searchable in Microsoft Purview.

2.1 Smart Data Search.

Once the metadata is ingested into the Microsoft Purview Data Map, it can be searched using Microsoft Purview’s Smart Data Searching in the Data Catalog.

Now you can use natural language description for data assets searching in the Microsoft Purview Catalog.

Go to the new Microsoft Purview Portal: https://purview.microsoft.com

Select the Data Catalog solution and then, Data Search.

Once you enter your search, Microsoft Purview returns a list of data assets and glossary terms that match the entered keywords, provided the user has data reader permissions for them.

In the example below, the search phrase was “I want to know the data related with diseases in Latam”.

You should know that the search returns all data assets in the collection(s) that best match the query. If a collection contains data assets that match the phrase, all scanned items are returned, even if some items do not match exactly.

The correspondence between search results and desired results depends on your Data Map design, the registered data sources, and the scope applied in scanning, which helps narrow down the most common searches in your business.

See for example a multiregional and business concepts as a design for your Data Map.

The Microsoft Purview relevance engine sorts through all the matches and ranks them based on what it believes their usefulness is to a user.

Many factors determine an asset’s relevance score, and the Microsoft Purview search team is constantly tuning the relevance engine to ensure the top search results have value to you.

2.2 Browse by applying filters.

Once you entered the search phrase and wait for a few seconds, you can see a Filters Pane where you can apply the following filtering criteria:

Asset Type
Data Source Type
Collection
Classification
Contact
Endorsement
Assigned Term
Sensitivity Label
Rating

Next figure shows the Asset Type filtering:

Filtering by “Data”, you can refine your search selecting one or more data asset types according to your referred data source:

Next figure shows the assets of type “Report”:

Then select any filter category you would like to narrow your results by and select any values you would like to narrow results to. For some filters, you can select the ellipses to choose between an AND condition or an OR condition, as next figure shows:

2.3 Curation process.

The process of contributing to the catalog by tagging, documenting, and annotating data sources that have already been registered is known as metadata curation [Metadata curation in Microsoft Purview | Microsoft Learn].

The curation process is facilitated by selecting one or more data assets returned in the search that are assumed to be the curator's responsibility.

For example, in the next figure we show two selected data assets:

By clicking on “View selected,” you can access a screen to start adding attributes to the data asset's metadata.

Click on “Bulk edit”:

Selecting an attribute, you can add new values, replace an existing value with another one or remove values.

You can add as many attributes as you need.

Depending on the attribute, you can manage the proper values.

Purview’s Copilot can help enrich metadata by suggesting additional context, classifications, and annotations based on the data asset's content and usage, as well as can ensure consistency in metadata definitions and standards across the organization, reducing discrepancies and improving data quality.

Selecting “Suggestions”, you can observe many derived suggestions based on your business concepts.

You should know that AI models use general internet knowledge base data so it will not return company specific or custom definitions or terms. All terms should be stewarded before being published to ensure that the term and definition aligns to company use and the specific knowledge about the term. Microsoft Purview Data Catalog Responsible AI FAQ (Preview) | Microsoft Learn

Any terms and definitions provided via suggestions should be reviewed and aligned with the company's specific language standards. When a term is selected and created, it will be in draft status, allowing the steward to complete and finalize the term before deciding to publish it.

Conclusions.

Smart data searching and automated metadata enrichment significantly enhance the cataloging process, making it more efficient, comprehensive, and insightful.

These advanced capabilities not only improve data discoverability and governance but also empower users with richer, more contextualized information, leading to more informed and effective decision-making.

Learn more:

Microsoft Purview collections architecture and best practices | Microsoft Learn

Scans and ingestion | Microsoft Learn

How to search the Data Catalog | Microsoft Learn

Discover data with natural language search - YouTube

Metadata curation in Microsoft Purview | Microsoft Learn.

Best practices for describing data in Microsoft Purview | Microsoft Learn

How to create and manage glossary terms (Preview) | Microsoft Learn

Curate your data with Business Concepts (youtube.com)

How to configure and manage data catalog access policies (Preview) | Microsoft Learn

Implementing Route Summarization in Azure VMware Solution

jasonmedina — Tue, 19 Nov 2024 12:52:32 GMT

What is Route Summarization?

Route summarization, also known as route aggregation, is a technique used in networking to combine multiple routes into a single, summarized route. This helps reduce the size of routing tables and simplifies the routing process.

Why Use Route Summarization in Azure VMware Solution?

Route summarization for Azure VMware Solution (AVS) is essential in the following scenario:

Route Tables with a 400 UDR Route Limitation: If you need to direct AVS workload segments through a Network Virtual Appliance (NVA) like Azure Firewall or a third-party firewall, you must create a User-Defined Route (UDR) for each AVS segment individually. This can quickly become cumbersome if your AVS environment has over 400 segments, as there is a 400 UDR route limitation per Route Table.

Route Summarization in NSX

In AVS, NSX provides network virtualization to create and manage virtual networks and security. Additionally, you can set up route summarization directly within NSX. NSX consists of two gateway routers: a Tier-1 and a Tier-0. The Tier-0 gateway connects to external networks and can summarize routes before advertising them to the physical network, thereby propagating the summarized routes back to Azure and on-premises. However, since Azure VMware Solution is a managed service, customers do not have Read/Write NSX permissions to modify configurations on the Tier-0 gateway. Therefore, any route summarization must be performed at the Tier-1 gateway level.

If you have contiguous Workload Segments connected to your NSX Tier-1 gateway, summarization becomes more straightforward. Otherwise, ensure that all summary routes comprehensively cover your Workload Segments to avoid any segments from losing connectivity. To enable route summarization, we need to suppress AVS from advertising specific routes and only advertise the summarized route. Therefore, it’s crucial that all summary routes cover all workload segments to prevent any loss of connectivity.

Note: When using the Tier-1 gateway for summarization, only Workload Segments can be summarized; the AVS /22 Management address cannot be summarized. However, with the Virtual WAN Route Maps feature (still in Public Preview at the time of this writing), you will be able to summarize both the /22 Management address block and Workload Segments. Once the Virtual WAN Route Maps feature becomes generally available, I will explore this topic further in a future blog post.

Scenario Overview

Using the topology illustrated below, I will guide you through the step-by-step process of deploying summarization from the NSX T1 gateway. In my scenario, I have a Virtual WAN Hub deployed, which includes an ExpressRoute Gateway. This gateway in the Hub-VNet connects to both the Azure VMware Solution (AVS) and On-Premises environments. Additionally, the Hub has a VNet peering to a Spoke VNet. There is also a Global Reach connection between AVS and On-Prem, ensuring connectivity between the two.

Note: While my example utilizes VWAN, the summarization steps and behavior remain consistent with those of a traditional hub-and-spoke topology.

In AVS, there are four workload segments. Each local segment in NSX is configured as a /24 subnet and is connected to the same Tier-1 gateway.

Segment 1: 192.168.100.0/24
Segment 2: 192.168.101.0/24
Segment 3: 192.168.102.0/24
Segment 4: 192.168.103.0/24

The goal is to stop advertising these four specific routes to both Azure and on-premises networks. Instead, we’ll only advertise the summary route 192.168.100.0/22, which covers all four segments.

Note: Route Summarization should not contain networks that are extended using HCX.

Before configuring Route Summarization

As indicated by the blue arrows, the four routes listed below are being advertised from AVS to the VWAN Hub ExpressRoute Gateway. These routes are propagated to both the VWAN Hub and the Spoke VNet. Additionally, the four routes are advertised to on-premises via Global Reach.

VWAN Hub Effective Routes before summarization

As highlighted below, I am currently learning the /24 routes on the VWAN Effective Routes from AVS.

Summarization Steps

1. Log into NSX and navigate to Networking > Tier-1 Gateways. Locate your Tier-1 Gateway where all your workload segments are connected. Click on the three dots (circled in red) and select Edit.

2. Scroll down and expand the Route Advertisement section.

Click the icon next to Set Route Advertisement Rules (circled in red).

3. Click Add Route Advertisement Rule
Create a name for your summary route. In my example, I used “Summary-Route.”
Add the summary route you want to advertise under Subnets. I used 192.168.100.0/22. Make sure to hit enter after typing in your summary route so it appears circled in blue as shown in the diagram.
Click Add then click Save.

4. Under the T1 Route Advertisement section, disable All Connected Segments & Service Ports as illustrated in the diagram below (circled in red).

IMPORTANT: Ensure all your connected segments are included in your summary route(s). Any connected segment not covered by a summary route will lose connectivity. For example, a summary route of 192.168.100.0/22 covers segments 192.168.100.0/24 to 192.168.103.0/24. If an additional segment is configured as 192.168.104.0/24, it would not be covered by the 192.168.100.0/22 summary route. Since specific workload segments are suppressed and only summary routes are advertised, the 192.168.104.0/24 segment would lose connectivity unless a summary route is created for it.

5. Click save

6. Ensure that you are now receiving the summary route(s) in Azure or from your on-premises environment if you are using Global Reach. As shown in the diagram below, NSX T1 in AVS will exclusively advertise the summarized route 192.168.100.0/22. This route will be propagated to both Azure and on-premises environments via Global Reach

VWAN Hub Effective Routes after summarization

As highlighted below, I am now learning the /22 summarized route on the VWAN Effective Routes from AVS.

Entra ID federation with Google Workspace

irina_kostina — Sun, 30 Nov 2025 21:41:11 GMT

Entra ID federation with Google Workspace

Google Workspace federation allows you to manage user identities in your Entra ID tenants while authenticating these users through Google. This can be beneficial if your company wants to use a single source of identities across different cloud platforms.

This article covers the scenario where your domain is already added and verified in Entra ID.

Requirements

To use Google Workspace federation, you need to ensure that your federated users are created in the Entra ID directory. This can be done through various methods such as auto-provisioning or using the Graph API.

Keep in mind, that the out-of-the box federation for Google only works for gmail.com personal accounts. In order to federate with work accounts managed in Google Workspace, you have to configure SAML IDP federation.

Important:

Prior to configuring the federation make sure that the domain is added under Verified domains in Entra ID. Pay attention that domain is not yet shown as "Federated"

Configuring SAML Federation on Google Workspace side

To configure SAML federation in Google Workspace, follow these steps:

1. Create a Web Application in Google Admin Panel:

Navigate to https://admin.google.com/
Go to Apps -> Web and mobile apps
Click Add app -> Search for apps
Search for "Microsoft Office 365" and install it.

2. Download Metadata:

After installing the app, go to the app details and download the metadata.
Save the IdP metadata file (GoogleIDPMetadata.xml) as it will be used to set up Microsoft Entra ID later.

3. Enable Auto-Provisioning:

Enable auto-provisioning inside the "Microsoft Office 365" app.

4. Configure Service Provider Details:

On the Service Provider details page:

Select the option Signed response.
Verify that the Name ID format is set to PERSISTENT.
Under SAML attribute mapping, select Basic Information and map Primary email to IDPEmail.

5. Enable the App for Users in Google Workspace:

Go to Apps -> Web and mobile apps -> Microsoft Office 365 -> User access.
Select ON for everyone and save the changes.

Adding Federation for SAML Provider in Entra ID

! Even though federation with external IDP is available via Entra ID or Azure Portal, you have to use PowerShell to create federation for verified domains.

Using the IdP metadata XML file downloaded from Google Workspace, modify the $DomainName variable in the following script to match your environment, and then run it in a PowerShell session:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser -Force Install-Module Microsoft.Graph -Scope CurrentUser Import-Module Microsoft.Graph $domainId = "<your domain name>" $xml = [Xml](Get-Content GoogleIDPMetadata.xml) $cert = -join $xml.EntityDescriptor.IDPSSODescriptor.KeyDescriptor.KeyInfo.X509Data.X509Certificate.Split() $issuerUri = $xml.EntityDescriptor.entityID $signinUri = $xml.EntityDescriptor.IDPSSODescriptor.SingleSignOnService | ? { $_.Binding.Contains('Redirect') } | % { $_.Location } $signoutUri = "https://accounts.google.com/logout" $displayName = "Google Workspace Identity" Connect-MGGraph -Scopes "Domain.ReadWrite.All", "Directory.AccessAsUser.All" $domainAuthParams = @{ DomainId = $domainId IssuerUri = $issuerUri DisplayName = $displayName ActiveSignInUri = $signinUri PassiveSignInUri = $signinUri SignOutUri = $signoutUri SigningCertificate = $cert PreferredAuthenticationProtocol = "saml" federatedIdpMfaBehavior = "acceptIfMfaDoneByFederatedIdp" } New-MgDomainFederationConfiguration @domainAuthParams

To verify that the configuration is correct, you can use the following PowerShell command:

Get-MgDomainFederationConfiguration -DomainId $domainId |fl

Test the federation

To test the federation, navigate to https://portal.azure.com and sign in with a Google Workspace account:

As username, use the email as defined in Google Workspace. The user is redirected to Google Workspace to sign in.
After Google Workspace authentication, the user is redirected back to Microsoft Entra ID and signed in.

Troubleshooting

If you configured federation after users were created in Entra ID, it is possible that users will get an error AADSTS51004

AADSTS51004: The user account XXX does not exist in the YYY directory. To sign into this application, the account must be added to the directory.

This error is most likely caused by property ImmutableId being incorrect.

For Google Workspace federation ImmutableId has to be set as a primary email adress of the user.

Follow these steps to update the ImmutableID:

Convert the federated user to a cloud-only user (update the UPN to a non-federated domain)
Update the ImmutableId
Convert the user back to a federated user

Here's a PowerShell example to update the ImmutableId for a federated user:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser -Force Install-Module Microsoft.Graph -Scope CurrentUser Import-Module Microsoft.Graph Connect-MgGraph -Scopes 'User.Read.All', 'User.ReadWrite.All' #1. Convert the user from federated to cloud-only Update-MgUser -UserId test@example.com -UserPrincipalName test@example.onmicrosoft.com #2. Convert the user back to federated, while setting the immutableId Update-MgUser -UserId test@example.onmicrosoft.com -UserPrincipalName test@example.com -OnPremisesImmutableId 'test@example.com'

Conclusion

In summary, Entra ID federation with Google Workspace allows seamless user identity management and authentication across different cloud platforms.

Hit me up in comments if this worked for you!

How to monitor applications by using OpenTelemetry on Azure Container Apps

tsunomur — Tue, 10 Sep 2024 03:49:44 GMT

Overview

There are multiple methods available to monitor applications deployed on Azure Container Apps. While the official documentation outlines these techniques, this article aims to organize them for better understanding.

Although this guide is focused on Java Spring Boot applications, the sections regarding Azure Container Apps are applicable to other programming languages and frameworks too.

How to Monitor Applications on Azure Container Apps

The following approaches can be employed to monitor applications running on Azure Container Apps:

Using Application Insights (Azure Monitor OpenTelemetry Distro)
Using the OpenTelemetry Collector of the Container Apps Environment
Using Dapr
Forwarding system logs/console logs to Log Analytics

We will explain these monitoring methods in more detail later in this article.

In the latter section of this blog, we have provided concise guidance on using Application Insights and OpenTelemetry to monitor applications. If you're not well-versed in this topic, please refer to “Monitoring Applications with Application Insights and OpenTelemetry” section.

Demo Application Specifications

For this article, we created a simple demo application that performs a quadrature operation on two GET request values and returns the result. The /calc endpoint takes two parameters, and the controller sequentially invokes addition, subtraction, multiplication, and division APIs.

The API endpoints are within one Java app, capable of self-calling or being deployed as separate Container Apps.

In this demo, Container Apps are deployed for the front end (app1) and each operation (add, sub, mul, div). Spring Boot's Controller in the front end calls each Container App endpoint.

1. Using Application Insights (Azure Monitor OpenTelemetry Distro)

This method requires incorporating the Application Insights agent either within the application's container or directly into the application itself. Refer Enable Azure Monitor OpenTelemetry for .NET, Node.js, Python, and Java applications for more details.

It doesn't depend on Azure Container Apps, so if you're already monitoring with Application Insights, no changes are needed. Naturally, you can use all the features available in Application Insights.

However, each application (each Container App) needs to specify the Application Insights agent or embed it in the code, which could increase the container image size and require connection string management.

Transaction view

Here's how a transaction and its exception appear.

You can observe that app1's Container App calls the add, sub, mul, div Container Apps. The division by zero result is displayed, showing where and what type of exception occurred.

You can see a larger image by clicking on each screenshot.

Metric

The metrics produced by the OpenTelemetry Meter are visible in Application Insights.

2. Using the OpenTelemetry Collector of the Container Apps Environment

In the Container Apps Environment, you can enable the OpenTelemetry Collector to receive data output by the OpenTelemetry SDK from Container Apps.

Similar to using Application Insights, this method also requires specifying or implementing the OpenTelemetry agent, but it eliminates the need to manage connection strings for each Container App by not using Application Insights.

Although you won't use Application Insights per Container App, you need to specify the Application Insights connection string per Container Apps Environment if you use it as an export destination.

This option is the easiest if you're already using OpenTelemetry. However, currently, metrics cannot be sent to Application Insights. The following table is on Collect and read OpenTelemetry data in Azure Container Apps (preview)

How to Enable OpenTelemetry collector

Run the following command while setting up the Container Apps Environment to enable this feature. Once activated, the OTEL_EXPORTER_OTLP_ENDPOINT environment variable will be automatically configured for the application.

Azure CLI

az containerapp env telemetry app-insights set -n aca-otelsample -g rg-opentelemetry --connection-string <Connection String> --enable-open-telemetry-traces true --enable-open-telemetry-logs true

Azure Portal

Transaction view

Similar to method #1, it displays the service call relationships. You can observe that the application sequentially calls the add, sub, mul, and div Container Apps.

3. Using Dapr

Using Container Apps with Dapr allows you to make requests between Container Apps, like http://localhost:3500/v1.0/invoke/checkout/method/checkout/100. The Dapr container, working as a sidecar in Container Apps, intercepts the request and directs it to the appropriate Container App.

Currently, Application Insights can receive traces of requests passing through Dapr. It leverages the Application Insights connection string configured in the Container Apps Environment. To enable this, you simply need to activate Dapr in Container Apps, without needing an Application Insights (OpenTelemetry) agent configuration. This method is the simplest if using Dapr is assumed.

It's important to note that Dapr can only monitor inter-service communication; it does not capture intra-service traces or metrics. This means activities such as an application within Container Apps accessing its own API endpoint or generating logs are not monitored.

However, if the API is built as a separate Container App and the communication between them uses Dapr, it will be tracked. To gather intra-service data, you must use Application Insights or OpenTelemetry agents as outlined in method #1 and #2.

Please note that you cannot use method #2 at the same time (an error will occur when setting the connection string). This limitation applies only to sending data to Application Insights and does not affect the use of Dapr itself.

How to enable Dapr based instrumentation

The command below sets up a Container Apps Environment. The flag --dapr-instrumentation-key is used to indicate that Dapr should utilize the Instrumentation key from Application Insights.

Azure CLI

az containerapp env create --name aca-otelsample --resource-group rg-opentelemetry --location japaneast --dapr-connection-string <Connection String>

Details of the command can be found at az containerapp env create.

Bicep

This Bicep code corresponds to the Azure CLI --dapr-instrumentation-key option. You can include a connection string within the properties section.

resource env 'Microsoft.App/managedEnvironments@2024-03-01' = {
    name: aca_env_name
    location: location
    properties: {
      appLogsConfiguration: {
        destination: 'log-analytics'
        logAnalyticsConfiguration: {
          customerId: la.properties.customerId
          sharedKey: la.listKeys().primarySharedKey
        }
      }
      zoneRedundant: false
      daprAIConnectionString: <Connection String> // <----- Setting the Application Insights connection string
      workloadProfiles: [
        {
          workloadProfileType: 'Consumption'
          name: 'Consumption'
        }
      ]
      peerAuthentication: {
        mtls: {
          enabled: false
        }
      }
    }
  }

Transaction view

In method #1 and #2, the front-end application (app1) sequentially invoked REST APIs for each quadratic operation. Conversely, in the method involving Dapr, each REST API call seems isolated. This occurs because Dapr handles communication between Container Apps without visibility into the internal processing of the application.

4. Forwarding System Logs/Console Logs to Log Analytics

Within the Container Apps Environment, you have the ability to forward both system and console logs to Log Analytics. Even though only console logs are collected (excluding events or metrics), this method is the simplest for outputting platform logs without altering the application's code or Container Apps settings.

The following document details the steps for activating logging in Container Apps to Log Analytics: Log storage and monitoring options in Azure Container Apps

How to view logs

As observed, without a corresponding ID, it becomes challenging to track application transactions based on the logs produced in Log Analytics.

Monitoring Applications with Application Insights and OpenTelemetry

For those new to Application Insights and OpenTelemetry, here is a brief introduction. If you are already knowledgeable about these topics, feel free to skip ahead to the Conclusion.

What Application Insights Can Do

Application Insights is an Azure-native service designed for application monitoring. This tool allows you to collect logs, metrics, and traces from your applications and visualize the data.

Refer to the Application Insights overview for a summary of the Application Insights.

Application Insights is an Azure service compatible with OpenTelemetry, the open standard for monitoring. It supports various language SDKs and can receive data from OpenTelemetry, enabling application monitoring without code changes. This service conveniently handles traces, logs, and metrics all in one place.

Here is a brief guide on how to monitor your Java application using Application Insights. There are two ways to utilize Application Insights from Java:

Specifying the Azure Monitor OpenTelemetry Distro as a Java agent
Integrating it directly in the source code

The agent method requires substituting the OpenTelemetry agent with the Azure Monitor OpenTelemetry Distro. Before launching your Java applications with the Java agent, you need to set up the connection string in an environment variable or in configuration file as illustrated below.

export APPLICATIONINSIGHTS_CONNECTION_STRING=<Your Connection String>
  java -javaagent:"path/to/applicationinsights-agent-3.5.3.jar" -jar myapp.jar

Refer to the "Enable using code" section of the following document for implementation details.

Using Azure Monitor Application Insights with Spring Boot

The Java agent for Application Insights acts as a wrapper around OpenTelemetry components, offering comparable capabilities. In addition, it includes extended implementations that leverage the features available in Application Insights. For further details, refer Why should I use the "Azure Monitor OpenTelemetry Distro"?

What OpenTelemetry Can Do

OpenTelemetry enables the collection of logs, metrics, and traces, which can be integrated with various compliant tools. For an overview, visit the official site.

Key points are:

You retain ownership of your data without vendor lock-in
You need to learn only one set of APIs and conventions.

Similar to Application Insights integration discussed above, you can add OpenTelemetry to a Java app in two ways

Using the OpenTelemetry agent as a Java agent via a command-line argument (codeless instrumentation)
Adding it directly into the source code (code-based instrumentation)

To use codeless instrumentation approach, just add the Java agent as a java command argument for seamless integration without modifying your code:

# If Application Insights is used, this OpenTelemetry Java agent can be replaced by an Application Insights agent as discussed earlier.
java -javaagent:./opentelemetry-javaagent.jar -jar myapp.jar

You will also need to create an OpenTelemetry instance using Java Agent if you want to send metrics and logs, using the following code:

    package com.example.otelsample;
  import io.opentelemetry.api.OpenTelemetry;
  import org.springframework.context.annotation.Bean;
  import org.springframework.context.annotation.Configuration;
  import io.opentelemetry.api.GlobalOpenTelemetry;
  
  @Configuration
  public class OpenTelemetryConfig {
      @Bean
      public OpenTelemetry openTelemetry() {
          return GlobalOpenTelemetry.get();
      }
  }

After that step, you can output logs, generate events, and create metrics with OpenTelemetry using the code below:

    // Start an OpenTelemetry span and send an event
  Span span = tracer.spanBuilder("CalcController").startSpan();
  span.addEvent("Start calc");
  
  // Output a log
  logger.info("Dummy log message");
  
  // Create a counter
  counter = meter.counterBuilder("calc.counter")
          .setDescription("How many times the calculator has been run.")
          .setUnit("runs")
          .build();
  counter.add(1);

For code-based instrumentation implementation, refer to the documentation at https://opentelemetry.io/docs/languages/java/instrumentation/#manual-instrumentation-setup

You need to collect and visualize data from OpenTelemetry. Open-source tools like Jaeger and Zipkin handle traces and logs, while Prometheus collects metrics that can be visualized with Grafana.

Using the OpenTelemetry Collector lets you gather data without embedding code in your application for each tool. It acts as an interface between your app and various tools. The demo architecture in the official documentation is easy to understand.

"Demo Architecture", OpenTelemetry.io, https://opentelemetry.io/docs/demo/architecture/

Applications send data to the OpenTelemetry Collector's endpoint (http://localhost:<port>/, where port is 4317 or 4318), and the backend endpoints are specified in the OpenTelemetry Collector's configuration.

For example, here is a configuration for the OpenTelemetry Collector:

receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  
  exporters:
    debug:
      verbosity: detailed
    otlp/jaeger:
      endpoint: jaeger-all-in-one:4317
      tls:
        insecure: true
    prometheus:
      endpoint: "0.0.0.0:8889"
      const_labels:
        label1: value1
    zipkin:
      endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
      format: proto
  processors:
    batch:
  
  extensions:
    health_check:
    pprof:
      endpoint: :1888
    zpages:
      endpoint: :55679
  
  service:
    telemetry:
      logs:
        level: DEBUG
        sampling:
          enabled: false
    extensions: [pprof, zpages, health_check]
    pipelines:
      traces:
        receivers: [otlp]
        exporters: [debug, zipkin, otlp/jaeger]
      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [debug, prometheus]
      logs:
        receivers: [otlp]
        exporters: [debug]

This configuration is based on "Configuration", OpenTelemetry.io, https://opentelemetry.io/docs/collector/configuration/

In the receivers, set the receiving endpoints, and in the exporters, configure the export destinations. By configuring services in the pipelines, you can receive traces at the OTLP endpoint and export to Zipkin.

OpenTelemetry avoids vendor lock-in, offering developers choices but requiring various tools. For example, you might use Jaeger for logs and Prometheus for metrics, each needing specific configurations.

Application Insights simplifies this by providing data collection to visualization in one service, negating the need to learn multiple services.

Conclusion

As demonstrated, there are multiple ways to monitor Azure Container Apps. We hope you find the method that fits your application needs and preferred implementation approach.

Leveraging dynamic few-shot prompt with Azure OpenAI

franklinlindemberg — Wed, 04 Sep 2024 21:35:38 GMT

Few-shot prompt is a technique used in natural language processing (NLP) where a model is given a small number of examples (or “shots”) to learn from before generating a response or completing a task. This approach is particularly beneficial because it allows the model to understand and adapt to new tasks with minimal data, making it highly efficient and versatile. By leveraging few-shot prompts, users can achieve high-quality results without the need for extensive training datasets, thus saving time and computational resources. Additionally, this method enhances the model’s ability to generalize from limited examples, leading to more accurate and contextually relevant outputs.

One challenge of using few-shot prompts is that as the number of examples increases, the prompt can become excessively large, leading to inefficiencies and potential performance issues. To address this, a dynamic few-shot prompt technique can be employed. In this approach, a comprehensive list of prompts is stored in a vector store, and the user’s input is matched against this vector store to identify the most relevant examples. By utilizing OpenAI embeddings alongside the vector store, this method ensures that only the most pertinent examples are included in the prompt, thereby optimizing its size and relevance. This dynamic technique not only maintains the efficiency and effectiveness of few-shot learning but also enhances the model’s ability to generate accurate and contextually appropriate responses.

Architecture

The diagram above shows the overall architecture of the solution. Let's break down each component:

Vector Store - This store will hold the few-shot prompt examples. It is indexed by each example's input and the content is the input/output pair
Embedding Model - This model is responsible for transforming the user input into a vector, which can be used to query the vector store
GPT Model - This is the model we will be using for chat completion, it's the one responsible for providing answers to the user

Use Case

To demonstrate how dynamic few-shot prompt can be used, let's consider a scenario where we have a chat completion that can handle 3 different types of tasks:

Display data in a table format
Classify texts
Summarize texts

For each of this tasks we want to provide some examples, in order to better explain the model what it should do.

One simple way of doing this is to provide all examples related to these tasks in the prompt itself. This strategy comes with a few downsides:

Information Overload: Too many examples can overwhelm the model, making it difficult to discern the main request.
Confusion: The model might get confused and generate responses that are off-topic or irrelevant.
Accuracy Issues: The focus on examples can detract from the accuracy of the response.
Cost: More examples mean more tokens to be processed by the model, which increases cost.

It's easy to see that this strategy doesn't scale in case we need to support other types of tasks, since it would require more examples.

On the other hand, if we use dynamic few-shot prompt, we just use the most relevant examples (for the given user input) in the prompt. For an example we can decide that we only want to use the top 3 most relevant examples in the prompt.

To implement this strategy we just need the steps below:

Define examples list
- Each example is an object that contains an input (user question example) and an output (assistant response for that question)
- ```
{"input": "<example_input>", "output": "<example_output>"
```
Index examples list in vector store
- Index key is the embedded example's input
Find most relevant examples
- Embed the user input using the same embedding used on the previous step
- Use the embedded user input to find the most relevant examples in the vector store
Add examples in the prompt
- Adds example's input as a user message and its output as an assistant message

In this solution we use:

the SemanticSimilarityExampleSelector class from langchain_core package since it already implements most of the steps mentioned above:
- Embed examples' input and index them into the vector store
- Define how many relevant examples should be returned
- Return relevant examples for a given user input
InMemoryVectorStore
- In-memory implementation of VectorStore using a dictionary
- The package langchain-community provides other vector store implementations, like Azure Search
AzureOpenAIEmbeddings
- Uses an openAI embedding, like text-embedding-ada-002
- The package langchain-community provides other embedding implementations, like Text2Vec

Code Implementation

The implementation of this solution is really simple. We only need 2 files: requirements.txt and main.py. Their contents can be found below:

requirements.txt

langchain-openai==0.1.21 azure-identity==1.17.1 numpy==2.0.1

main.py

# imports from azure.identity import DefaultAzureCredential from openai import AzureOpenAI from langchain_openai import AzureOpenAIEmbeddings from langchain_core.example_selectors import SemanticSimilarityExampleSelector from langchain_core.vectorstores import InMemoryVectorStore # constants AZURE_OPENAI_ENDPOINT = "<AZURE_OPENAI_ENDPOINT>" DEPLOYMENT_EMBEDDING = "<DEPLOYMENT_EMBEDDING>" DEPLOYMENT_CHAT = "<DEPLOYMENT_CHAT>" OPENAI_API_VERSION = "2023-05-15" SYSTEM_MESSAGE = 'You are a helpful assistant that uses previous interactions to determine how a questions should be answered. You should try to use those same response formats' # the examples to be indexed in the vector store. Each example should be in the format `{"input": "<example_input", "output": "<example_output>"}` EXAMPLES_LIST = [ # data examples { "input": "Can you provide the population of the top 5 most populous countries?", "output": ''' | Country | Population (2023) | |---------------|-------------------| | China | 1,425,000,000 | | India | 1,417,000,000 | | United States | 332,000,000 | | Indonesia | 276,000,000 | | Pakistan | 231,000,000 | ''' }, { "input": "What are the top 5 programming languages in 2023 by popularity?", "output": ''' | Rank | Programming Language | Popularity (%) | |------|----------------------|----------------| | 1 | Python | 29.9 | | 2 | JavaScript | 19.1 | | 3 | Java | 16.2 | | 4 | C# | 8.3 | | 5 | PHP | 6.1 | ''' }, { "input": "Can you list the top 5 highest-grossing movies of all time?", "output": ''' | Rank | Movie Title | Gross Revenue (USD) | |------|-------------------------|---------------------| | 1 | Avatar | $2.923 billion | | 2 | Avengers: Endgame | $2.798 billion | | 3 | Titanic | $2.195 billion | | 4 | Star Wars: The Force Awakens | $2.068 billion | | 5 | Avengers: Infinity War | $2.048 billion | ''' }, { "input": "What are the top 5 universities in the world according to the 2023 QS World University Rankings?", "output": ''' | Rank | University | Location | |------|-----------------------------------|----------------| | 1 | Massachusetts Institute of Technology (MIT) | USA | | 2 | University of Cambridge | UK | | 3 | Stanford University | USA | | 4 | University of Oxford | UK | | 5 | Harvard University | USA | ''' }, { "input": "Can you provide the GDP of the top 5 largest economies in 2023?", "output": ''' | Rank | Country | GDP (USD Trillions) | |------|---------------|---------------------| | 1 | United States | 25.3 | | 2 | China | 17.7 | | 3 | Japan | 4.9 | | 4 | Germany | 4.2 | | 5 | India | 3.5 | ''' }, # Classification examples { "input": "Classify the following text: 'The quick brown fox jumps over the lazy dog.'", "output": "Sentence" }, { "input": "Classify the following text: 'To be, or not to be, that is the question.'", "output": "Quote" }, { "input": "Classify the following text: 'Once upon a time, in a land far, far away, there lived a young princess.'", "output": "Story" }, { "input": "Classify the following text: 'E=mc^2 is a formula expressing the relationship between mass and energy.'", "output": "Scientific Statement" }, { "input": "Classify the following text: 'I hope you have a great day!'", "output": "Greeting" }, # Summarization examples { "input": "Summarize the following paragraph: 'The rapid advancement of technology has significantly impacted various industries. Automation and artificial intelligence are transforming the workforce, leading to increased efficiency but also raising concerns about job displacement. Companies are investing heavily in tech to stay competitive, while governments are grappling with the need to update regulations.'", "output": ''' - Technology is transforming industries through automation and AI. - Increased efficiency comes with concerns about job displacement. - Companies invest in tech; governments update regulations. ''' }, { "input": "Summarize the following paragraph: 'Climate change is one of the most pressing issues of our time. Rising global temperatures are causing more frequent and severe weather events, such as hurricanes and wildfires. Efforts to mitigate climate change include reducing carbon emissions, transitioning to renewable energy sources, and protecting natural habitats.'", "output": ''' - Climate change leads to severe weather events. - Mitigation efforts focus on reducing carbon emissions and using renewable energy. - Protecting natural habitats is crucial. ''' }, { "input": "Summarize the following paragraph: 'The education system is undergoing significant reforms to better prepare students for the future. Emphasis is being placed on critical thinking, problem-solving, and digital literacy. Schools are incorporating more technology into the classroom and offering courses that align with the demands of the modern workforce.'", "output": ''' - Education reforms focus on future readiness. - Key skills: critical thinking, problem-solving, digital literacy. - More technology and modern workforce-aligned courses in schools. ''' }, { "input": "Summarize the following paragraph: 'The healthcare industry is seeing a shift towards personalized medicine. Advances in genetic research allow for treatments tailored to individual patients, improving outcomes and reducing side effects. This approach is particularly promising in the treatment of cancer and rare genetic disorders.'", "output": ''' - Healthcare is moving towards personalized medicine. - Genetic research enables tailored treatments. - Promising for cancer and rare genetic disorders. ''' }, { "input": "Summarize the following paragraph: 'Remote work has become increasingly common, especially after the COVID-19 pandemic. Many companies have adopted flexible work policies, allowing employees to work from home or other locations. This shift has led to changes in workplace dynamics, with a greater emphasis on digital communication and collaboration tools.'", "output": ''' - Remote work is more common post-COVID-19. - Companies adopt flexible work policies. - Emphasis on digital communication and collaboration tools. ''' } ] # get default azure credentials. You might need to run `az login` first. credential = DefaultAzureCredential() # get an openai token using the default credentials. Used to authenticate with openai openai_token = credential.get_token("https://cognitiveservices.azure.com/.default").token # create openai client client = AzureOpenAI( azure_endpoint=AZURE_OPENAI_ENDPOINT, azure_deployment=DEPLOYMENT_CHAT, azure_ad_token=openai_token, api_version=OPENAI_API_VERSION ) # create openai embeddings client embedding = AzureOpenAIEmbeddings( azure_endpoint=AZURE_OPENAI_ENDPOINT, deployment=DEPLOYMENT_EMBEDDING, azure_ad_token=openai_token, openai_api_version=OPENAI_API_VERSION ) # creates an example selector that uses semantic similarity to retrieve examples example_selector = SemanticSimilarityExampleSelector.from_examples( EXAMPLES_LIST, # list of examples to index embedding, # embedding model to use InMemoryVectorStore, # vector store to use k=3, # number of examples to retrieve input_keys=["input"], # keys in the examples that contain the input text ) while True: # gets user input user_input = input("Enter a sentence: ") # select the most relevant examples based on the user input res = example_selector.select_examples({"input": user_input}) messages = [ {"role": "system", "content": SYSTEM_MESSAGE}, ] print('retrieved examples:') for ex in res: # adds each example input as a user message and the corresponding output as an assistant message messages.append({"role": "user", "content": ex['input']}) messages.append({"role": "assistant", "content": ex['output']}) print('question:{}\nanswer:{}'.format(ex['input'], ex['output'])) # adds the user input as a user message messages.append({"role": "user", "content": user_input}) # create a completion using the messages response = client.chat.completions.create( messages=messages, model="gpt-4o", ) # print the response from the model print('response: ', response.choices[0].message.content)

Code Execution

Pre-Requisites

Python 3.10
An OpenAI GPT deployment (we suggest gpt-4o model)
An OpenAI Embedding deployment (we suggest text-embedding-ada-002 model)
Have permission on the OpenAI service (more details here)

Install dependencies

pip install -r requirements.txt

Run

python main.py

Below you can see the results of running the code with some different types of input. Notice that on the retrieved examples section it only shows 3 examples, which are the most relevants for that input.

Enter a sentence: list me the biggest airplanes retrieved examples: question:Can you list the top 5 highest-grossing movies of all time? answer: | Rank | Movie Title | Gross Revenue (USD) | |------|-------------------------|---------------------| | 1 | Avatar | $2.923 billion | | 2 | Avengers: Endgame | $2.798 billion | | 3 | Titanic | $2.195 billion | | 4 | Star Wars: The Force Awakens | $2.068 billion | | 5 | Avengers: Infinity War | $2.048 billion | question:Can you provide the GDP of the top 5 largest economies in 2023? answer: | Rank | Country | GDP (USD Trillions) | |------|---------------|---------------------| | 1 | United States | 25.3 | | 2 | China | 17.7 | | 3 | Japan | 4.9 | | 4 | Germany | 4.2 | | 5 | India | 3.5 | question:Can you provide the population of the top 5 most populous countries? answer: | Country | Population (2023) | |---------------|-------------------| | China | 1,425,000,000 | | India | 1,417,000,000 | | United States | 332,000,000 | | Indonesia | 276,000,000 | | Pakistan | 231,000,000 | response: Certainly! Here are some of the biggest airplanes in terms of size and capacity: | Rank | Aircraft | Description | |------|---------------------------------|-----------------------------------------------------------------------------------------------------------| | 1 | Antonov An-225 Mriya | The largest cargo aircraft ever built, featuring 6 engines and a maximum takeoff weight of 640 tons | | 2 | Airbus A380 | The largest passenger airliner, with a double-deck configuration and capacity for up to 853 passengers | | 3 | Boeing 747-8 | One of the largest commercial aircraft, known as the "Queen of the Skies," with a capacity of 410-524 passengers | | 4 | Lockheed C-5 Galaxy | A large military transport aircraft used by the U.S. Air Force, with a maximum takeoff weight of 381 tons | | 5 | Boeing 777-9 | The newest variant of the Boeing 777 series, featuring a longer fuselage and higher capacity, accommodating up to 426 passengers | These aircraft are recognized for their impressive sizes and capabilities, serving various purposes from cargo transport to long-haul passenger flights. Enter a sentence: classify the following text: F=ma is Newton's second law of motion retrieved examples: question:Classify the following text: 'E=mc^2 is a formula expressing the relationship between mass and energy.' answer:Scientific Statement question:Classify the following text: 'The quick brown fox jumps over the lazy dog.' answer:Sentence question:Classify the following text: 'To be, or not to be, that is the question.' answer:Quote response: Scientific Statement Enter a sentence: summarize the following text: Exercise offers numerous benefits that significantly enhance one’s quality of life. Regular physical activity helps maintain a healthy weight, reduces the risk of chronic diseases such as heart disease, diabetes, and certain cancers, and improves cardiovascular health. Additionally, exercise boosts mental health by reducing symptoms of depression and anxiety, enhancing mood, and improving sleep quality. It also increases energy levels, strengthens muscles and bones, and promotes better flexibility and balance. Overall, incorporating exercise into your daily routine can lead to a longer, healthier, and more fulfilling life. What kind of exercise do you enjoy the most? retrieved examples: question:Summarize the following paragraph: 'The healthcare industry is seeing a shift towards personalized medicine. Advances in genetic research allow for treatments tailored to individual patients, improving outcomes and reducing side effects. This approach is particularly promising in the treatment of cancer and rare genetic disorders.' answer: - Healthcare is moving towards personalized medicine. - Genetic research enables tailored treatments. - Promising for cancer and rare genetic disorders. question:Summarize the following paragraph: 'The education system is undergoing significant reforms to better prepare students for the future. Emphasis is being placed on critical thinking, problem-solving, and digital literacy. Schools are incorporating more technology into the classroom and offering courses that align with the demands of the modern workforce.' answer: - Education reforms focus on future readiness. - Key skills: critical thinking, problem-solving, digital literacy. - More technology and modern workforce-aligned courses in schools. question:Classify the following text: 'E=mc^2 is a formula expressing the relationship between mass and energy.' answer:Scientific Statement response: - Exercise greatly enhances quality of life. - Benefits: healthy weight, reduced chronic disease risk, improved cardiovascular and mental health, better sleep, increased energy, stronger muscles and bones, and improved flexibility and balance. - Leads to a longer, healthier, and more fulfilling life. - What kind of exercise do you enjoy the most?

Next Steps

Now that you are familiar with dynamic few-shot technique you can try to take it to the next level by:

Use a different vector store
Use a different embedding model
Update the examples/prompt so it can support a new task

Conclusion

The dynamic few-shot prompt technique represents a significant improvement over traditional few-shot learning. By leveraging a vector store and an embedding model, this method ensures that only the most relevant examples are included in the prompt, optimizing its relevance, size and ultimately cost. This approach not only maintains the efficiency and effectiveness of few-shot learning but also enhances the model’s ability to generate accurate and contextually appropriate responses. As a result, users can achieve high-quality outcomes with minimal data, making this technique a powerful tool for a wide range of applications.

References

Langchain SQL Agent - Using dynamic few-shot prompt

Microsoft Fabric Metadata Driven Pipelines with Mirrored Databases

jehayes — Mon, 26 Aug 2024 21:59:25 GMT

Microsoft Fabric's database mirroring feature is a game changer for organizations using Azure SQL DB, Azure Cosmos DB or Snowflake in their cloud environment! Mirroring offers near real-time replication with just a few clicks AND for at no cost!

Features:

No data movement cost when mirroring
No storage cost for mirrored tables
No consumption of Fabric Capacity Units
Mirror all tables in your source database or just a few, with the capability to add more tables as your Fabric analytics environment grows
Source data continuously replicated with no data pipelines to configure
Data landed in delta tables in One Lake which are optimized by default
SQL endpoint and default semantic model automatically created

What can you do?

Run near real-time queries against the SQL endpoint with no impact to your source system since the data is replicated
Share the mirrored database across your Fabric tenant
Run cross database queries from within the mirrored database SQL endpoint or from One Lake when the mirrored database is shared
Create SQL views over mirrored data, which can include joins or unions with data from other mirrored databases, warehouses or lakehouse SQL endpoints
Configure row-level security and object level security against the mirrored data
Create Direct Lake near real-time Power BI reports against the default semantic model or against a new model
Copy mirrored data into a lakehouse and used in Spark notebooks or Data Science workloads
And… build faster, simpler metadata driven pipelines! Which is the focus of this article!

Metadata Driven Pipelines with Microsoft Fabric Mirrored Databases

Metadata-driven pipelines in Microsoft Fabric enable you to streamline data ingestion and transformations with minimal coding, lower maintenance, and enhanced scalability. And when your source is Azure SQL DB, Azure Cosmos, or Snowflake, this becomes even easier!

Architecture Overview

With Mirroring, the data is replicated and readily available in OneLake. This eliminates the pipeline which brings data into Fabric. After mirroring is configured:
1. A SQL endpoint is automatically created, allowing you and your users to query the data in near real time without impacting source data performance
2. A default semantic model is also automatically created, allowing you and your users to create near real time Power BI reports. However, best practice is to create a new semantic model, which provides more features and options than the default semantic model
A Fabric Data Warehouse is created to contain:
1. The fact and dimension tables (star schema) which simplifies and optimizes the analytical data model for SQL querying and semantic models
2. SQL views and stored procedures used in data transformations
3. The metadata table which holds the information on how to transform load each fact or dimension table
The Fabric Data Pipelines are created and scheduled to perform:
1. A Lookup activity on the metadata table to get the information on how to load each fact or dimension table in the Fabric Data Warehouse
2. Copy Data activities for full loads with the source being a SQL view over the mirrored tables and the destination a table in the Fabric Data Warehouse
3. Stored Procedure activities for incremental loads to merge the latest data from the mirrored data source into the Data Warehouse destination table
A default semantic model is automatically created over the Fabric Data Warehouse. But create a new one per best practices
Build Power BI reports for analytical reporting

Why 2 semantic models? Fabric Data Warehouse vs Mirrored Database SQL Endpoint for semantic model reporting

For this architecture, I considered using:

Just the mirrored tables
SQL views over the mirrored tables
A new Fabric Data Warehouse with data loaded from the mirrored tables

Since SQL views over mirrored tables always resort to Direct Query rather than Direct Lake, I decided against building a semantic model over SQL views. Instead, I created a semantic model over the mirrored tables and a separate Fabric Data Warehouse and semantic mode over it.

The semantic model over the mirrored tables:

Is used only by users who understand the source data schema
Allows for near-real time access for data to answer ad-hoc questions that are not analytical in nature such as the order shipment status of a particular order number or current stock availability of a specific product at a certain location
Incur no data storage cost but could be offset by consumption of Capacity Units if reports/queries are complex
Can leverage Power BI Direct Lake connection for faster report performance if the report is not too complex; if the query is too complex, it will resort to Direct Query
Could include other lakehouses, mirrored databases, or data warehouse tables in the SQL endpoint and thus the semantic model but Power BI reports using these tables will always resort to direct query rather than direct lake connect
Can include complex relationships between tables and unexpected results may be returned if the semantic model and/or reports are not configured correctly

The semantic model over the Fabric Data Warehouse:

Requires scheduled data refreshes but will be relatively fast since the source data is already in Fabric
Is best for more analytical questions such as "What was our sales revenue by month by customer location?" or "What is our days on hand for products in a particular shipping warehouse?"
Allows for user friendly data warehouse table and column names rather than using the potentially cryptic mirrored database table and column names
Eliminates snowflake schema, allowing for better performing reports and delivery of consistent results without having to understand complex relationships and filtering rules
Is more likely to leverage Direct Lake connection in Power BI reports since model is simpler
Allows other data sources to be loaded into the same warehouse, eliminating cross database joins in reports that automatically resort to direct query
Leverages a simpler star schema model resulting in faster reports with less consumption of capacity units

This solution addresses two key use cases: providing near real-time responses to queries about specific data transactions and statuses, and delivering rapid analytics over large datasets. Continue reading to learn how to implement this architecture in your own environment.

Solution details

Below are detailed steps to build the metadata pipeline. The data source is the Wide World Importers SQL database, which you can download here. Then follow the instructions to import into an Azure SQL DB.

Configure database mirroring

From the Synapse Data Warehouse experience, choose the Mirrored Azure SQL DB Option:

Then choose the tables to mirror:

After the mirroring has started, the main canvas will say “Mirrored Azure SQL Database is running’; Click on Monitor replication to see the number of rows replicated and the last completion time:

At this point, both a SQL analytics endpoint and default semantic model are created (1a). But I created a new semantic model(1b) and set up the table relationships:

2. Create a Fabric Data Warehouse

Create the fact tables or any other tables that will be incrementally loaded (2a). You can also manually create the dimension tables or tables are full loaded OR you can specify to auto-create the tables in the pipeline Copy Data activity, as I do later.

Create the views over the mirrored database tables and stored procedures (2b):

Create and load the metadata table (2c) with information on how to load each fact or dimension table:

3. Create the data pipelines to load the fact and dimension tables

Below is the orchestrator pipeline:

Set variable – set pipelinestarttime to the current date/time. This is logged in the metadata driven pipeline table for each table

Lookup – get the table load attributes from metadata table for each table to load

For each table to load

Invoke the pipeline, passing in the current row object and the date/time the orchestrator pipeline started:

Load warehouse table pipeline

Set variable pipeline start time for tracking the time of each table load

If activity – check if full or incremental load

If full load

Use Copy Data Activity to load the Data Warehouse table, set the pipeline end time variable and update the metadata table with load information

Copy data activity

Source settings reference the view over the mirrored database:

Destination settings reference data warehouse table:

Note that the data warehouse table will be dropped and re-created each time

Set the pipeline end time variable

Run Script to update the pipeline run details for this table

If not a full load, then run the incremental load activities

Lookup activity calls a stored procedure to insert or update new or changed records into the destination table. The value for the StartDate parameter is the latest date of the previous load of this table. The value for the EndDate parameter is usually a null value and only set if there is a need to load or reload a subset of data.

The stored procedure performs an insert or update, depending upon whether or not the key value exists in the destination. Only the records from the source that have changed since the last table load are selected. This reduces the number of updates performed.

The stored procedure returns how many rows were inserted or updated, along with the latest transaction data of the data loaded, which is needed for the next incremental load.

Set the pipeline end time

Script activity updates the table load details:

4. Build a new semantic model in the Fabric/Power BI service

Create relationships between the tables, DAX calculations, dimension hierarchies, display formats – anything you need for your analytics

Note that all tables have Direct Lake connectivity as noted by the dashed, blue line. Direct Lake has the performance of Import semantic models without the overhead of refreshing the data.

5. Create reports

Create reports from your semantic model

Continue on building out more reports and dashboards, setting up security, scheduling data warehouse refresh (which will now be super fast since the source data is already in Fabric), creating apps, adding more data sources – whatever it takes to get the analytics your organization needs into Fabric!

Mirroring - Microsoft Fabric | Microsoft Learn

What is data warehousing in Microsoft Fabric? - Microsoft Fabric | Microsoft Learn

Data Factory in Microsoft Fabric documentation - Microsoft Fabric | Microsoft Learn

Work with semantic models in Microsoft Fabric - Training | Microsoft Learn

Create reports in the Power BI - Microsoft Fabric | Microsoft Learn

Dimensional modeling in Microsoft Fabric Warehouse - Microsoft Fabric | Microsoft Learn

If your source database is not supported for mirroring (yet!), check out these other articles I wrote:

Metadata Driven Pipelines for Microsoft Fabric - Microsoft Community Hub

Metadata Driven Pipelines for Microsoft Fabric – Part 2, Data Warehouse Style - Microsoft Community Hub

Migrating from Azure APIM STv1 to STv2: New Options and Considerations

srinipadala — Fri, 23 Aug 2024 22:06:40 GMT

As the support for Azure API Management (APIM) STv1 platform ends on August 31, 2024, it’s crucial for customers to migrate their instances to the STv2 platform. This blog will focus on the new migration options introduced to facilitate this process, as outlined in the attached document.

Why Migrate to STv2?

With the end of support for STv1, instances on this platform will no longer have a Service Level Agreement (SLA). Migrating to STv2 ensures continued support and access to the latest features and improvements in Azure APIM.

New Migration Options

Over the past year, several limitations in the migration process have been addressed to make it easier for instances injected into a virtual network. Here are the key improvements:

Portal Experience: Enhanced user interface for a smoother migration process.
Public IP Optional: The service can now provision a managed IP, making the public IP optional.
Retain Old Gateway: Ability to keep the old gateway for a longer duration for validation purposes.
Release Old Subnet: Option to release the old subnet sooner for customers who need to revert.

Networking Dependencies

One of the biggest challenges in the migration process has been networking dependencies, particularly the need for new subnets and IP changes. The latest migration option addresses this by allowing the retention of original IPs, both public and private.

Key Considerations for the New Migration Option

Subnet Capacity: The subnet must have enough capacity to accommodate the STv2 instance. This means the subnet should be at least half empty to allow the creation of a new STv2 gateway alongside the STv1 gateway.
Coexistence of Instances: If the subnet contains other APIM instances, they should be migrated as soon as possible to avoid conflicts during scaling or update operations.
Subnet Delegation: The subnet cannot have any deployed or delegated resources. Ensure that any delegations are removed ahead of the migration.
Disable Scaling Rules: To prevent issues during the migration, disable all scaling rules. The default coexistence period is 15 minutes for external VNet injected instances and 4 hours for internal VNet injected instances. If multiple STv1 instances exist in the same subnet, disable scaling rules across all instances until migration is complete.
Networking Settings: STv2 requires additional networking settings compared to STv1. Ensure that traffic to Azure is allowed in the existing Network Security Group (NSG), Network Virtual Appliances (NVAs), and other networking controls. This includes:

Adding an outbound rule to Azure KeyVault in the NSG.
Adding a service endpoint to Azure KeyVault on the subnet if force tunneling through an NVA.
Allowing traffic to Azure KeyVault from the subnet address space at the NVA.

Migration Options Within the Same Subnet

Preserve Original IP Addresses: This option retains the original public and private IPs, but involves downtime while the IPs are transferred from the old gateway to the new gateway.
New IP Addresses: This option uses new public IPs, which are pre-created to allow for network dependency adjustments and communication with partners. It also allows specifying the retention time of the old gateway for internal VNet injected instances, providing extended time for validation and updating network dependencies.

Migration Process

The migration process involves creating a new STv2 gateway alongside the existing STv1 gateway in the same subnet. Here are the detailed steps:

Create New Gateway: The migration process creates a new STv2 gateway in the same subnet as the old gateway. The old gateway continues to handle traffic using custom DNS settings.
Preserve IP Option:

The IPs are transferred from the old gateway to the new gateway, resulting in a brief downtime as the IPs will not respond to traffic.
The old gateway is deleted after the migration is successful.

New IP Option:

Pre-created public IPs are assigned to the new gateway.
The old gateway is retained for 15 minutes for external VNet injected instances and 4 hours for internal VNet injected instances.
Validation activities and DNS updates can be performed during the retention period to achieve a no-downtime migration.
The old gateway is deleted after the retention period elapses.

Additional Considerations

Infrastructure Configuration Lock: The infrastructure configuration will be locked for the entire duration of the migration.
Downtime: The preserve IP option will have a downtime during the IP transfer. The new IP option avoids this downtime by using pre-created public IPs.
Networking: NSG with the rules for stv2 is required to be attached to the subnet. Existing subnets need to have the additional outbound rule for Azure Key Vault
Multi Regions: There is no option to selectively upgrade the locations. A single operation will orchestrate upgrading all the regions one at a time.

Verification and Monitoring

Verify Networking: The new UI includes a Verify button to check if the network meets the requirements. This static check looks for NSGs, service endpoints, and DNS configurations but does not check blocks at the NVA level, which need to be verified manually.
Diagnose and Solve Problems: Additional detectors are available to monitor the status of the migration.

Conclusion

Migrating from STv1 to STv2 is essential to ensure continued support and access to the latest features in Azure APIM. The new migration options significantly simplify the process by addressing key challenges, particularly networking dependencies. By following the considerations and steps outlined above, customers can achieve a smooth and successful migration.

Refer the learn documentation for the complete list of migration options including the same subnet migration using Azure CLI.

!!! Note: The portal experience is not completely available and should be released soon.

Feel free to reach out with any questions or for further assistance with your migration process!

I hope this blog helps you understand the new migration options and considerations for moving from Azure APIM STv1 to STv2. If you have any specific questions or need further details, let me know!

Explaining Purview concepts: Domains, Business Domains, Collections, Data Products and Data Assets.

Eduardo_Noriega — Wed, 21 Aug 2024 14:36:20 GMT

Microsoft Purview offers a comprehensive suite of tools for governing your organization's data through the solutions included in Purview Unified Platform.

To catalog your data assets, you must first define your data map, which is composed of collections. However, you might encounter several related concepts that can be confusing when trying to streamline your organization’s data governance.

In this article, we will explain the following concepts: Domains, Business Domains, Collections, Data Products, and Data Assets. We will also provide examples of how to use them correctly in your organization.

Why “domains” and “business domains” in Purview?

In the Classic Purview experience, an organization can have multiple accounts in its tenant.

The Unified Purview Portal is intended to manage a single Microsoft Purview resource with multiple domains for your organization, and multiple accounts don’t exist in a single tenant.

The goal is to replace multiple tenant accounts with multiple domains within a single tenant.

Domains will continue to address the problems that accounts solve today and will also serve as the container for the collections and assets.

Every Microsoft Purview Data Map starts with a default domain.

Next figure shows a structure for a domain in the new Microsoft Purview Portal/Data Map.

The +New domain option appears as disabled for many users, but if you are a Purview Administrator you can add up to four more custom domains in your Microsoft Purview Data Map. How to manage domains and collections | Microsoft Learn

When you add a new domain, you can add new collections as well:

In above figure, you can see a Customer Service Domain and an Operations domain, in addition to the default domain.

New collections can be added under the custom domains, as in the default domain were.

The new experience uses a single, primary Microsoft Purview account, that represents a tenant-level/organization-wide account. In our example, this is “FabricPurviewDemo”, the name of our Purview Resource subscription, for managing our tenant towards unifying organization's governance, policy, compliance, risk, and security.

If you already have multiple accounts in the classic experience, you'll select a primary account when you upgrade to the new experience. Get ready for the next enhancement in Microsoft Purview governance solutions | Microsoft Learn

After you upgrade an existing account to the new experience, all other Microsoft Purview accounts in your tenant will continue to be accessed via the classic portal. Fine grained access control via roles and permissions at collection scopes will continue to function as-is after your accounts are upgraded. In addition, there are new tenant-level roles that can be managed in the new portal.

The Data Map view looks like the next figure shows, with three domains:

This Data Map contains collections under the default domain (FabricPurviewDemo).

We have three collections for scanning data assets in the desired sources, they are: Diseases, Human Resources and Finance. You can only scan one data source into a single collection.

In this figure you can notice Fabric as a data source for the Diseases Collection.

Human Resources and Finance are collections for scanning other data sources.

Operations and Customer Services are domains at the same level as the default domain and they don’t have collections added yet.

Business Domains

While collections are intended to accept data assets directly from data sources through scanning processes, which is meaningless to business owners, business domains provide boundaries where governance policies reach data products. The goal is to empower an enterprise domain owner to manage their data products and concepts and to establish rules for their access, use, and distribution. With this goal in mind, you could establish many types of business domains:

Fundamental business areas - human resources, sales, finance, supply chain, etc.

Overarching subject areas - product, parts, etc.

Boundaries based on organizational functions - customer experience, cloud supply chain, business intelligence, etc.

Business Domains in Microsoft Purview provide a way to curate the data assets that are scanned in the Data Map solution. Curate your data with Business Concepts (youtube.com)

Go to Data Catalog. Under Data Management, select Business Domains.

Next figure shows several Business Domains defined in our organization.

Figure above shows the use of Data estate mappings to assign collections to the selected Business Domain. You can select more than one collection to manage the scanned data assets of the selected collections as part of this business domain.

You can further move data assets between business domains as needed.

Some data sources permit scanning with a scoped approach, while others mandate full scans. Scoped scanning simplifies assigning collections to business domains, reducing the need to relocate assets, as full scans import a larger set of data assets.

Details tab shows the Name, Type, Owner, Status (published, draft), Data quality score and Health actions about this business domain:

The Roles tab allows to define business domain owners, business domain readers, data products owners, data steward, data catalog readers, data quality stewards, data quality readers and other profiles.

To create a Business Domain, press the +New Business domain button. Then you will see a window like that:

A business domain must have a name, a description, a type (Functional Unit, Line of business, Data Domain, Regulatory or Project).

“Finance” and “Human Resources” are of type “Functional Unit” and are intended to group data assets discovered from Azure Databases containing several master tables about our clients, contracts, workers, incomes and expenses.

Our business domain "Diseases" is of type "Data Domain" and is intended to group data assets discovered in our Fabric tenant, which contains some reports, semantic models, data pipelines, two data warehouses, and a Lakehouse. All these data assets will be maintained with policies and roles to control access within our organization and allow stakeholders to manage them using medical sciences terms.

On your business domain's detail page, you can see the Business concepts section. Here you can see your data products, glossary terms, objective and key results (OKRs), critical data elements and custom attributes. You can go through the cards to create all of them about a specific business domain.

Let's continue with these concepts and some examples about them.

Data Products

A data product is a set of data assets discovered in one or more data sources that serve a meaningful business purpose and support end users' specific needs.

Managing data as a product offers numerous benefits for both businesses and users, among these are: facilitate cross-functional collaboration and reduce the time and effort to find, process and analyze data by owners.

The data product provides context for these assets, grouping them under a use case for data consumers. A business domain can house many data products, but a data product is managed by a single business domain. 

Next figure shows the details about the Diseases Business Domain at the right.

Selecting Go to data products and + New data product you can define as many Data Products as you need for that business domain. How to create and manage data products (Preview) | Microsoft Learn

Next figure shows adding a new Data Product in the Diseases Business Domain.

In this tab you must define a name, a description, the type and the owner. The selected type of our data product can be “Dataset”, according to the type of data assets this data product will contain.

In the second tab you must define Business Details.

At this point you can follow a wizard to define the purpose and other details about this data product. The last wizard screen allows you to add data assets from the assigned collections or define the access policy.

Next figure shows adding data assets:

You can filter data by type on the left and add the desired data assets by checking them.

Depending on the type, Microsoft Purview finds all data assets of the type defined for the Data Product. You can apply filters and move through pages to select the appropriate data assets.

Next figure shows the Data Product created in the Diseases Business Domain:

On the same screen, you can observe three data assets, and five glossary terms added to this data product. Glossary terms are defined for the entire business domain. You must select the appropriate terms from this set to be used by users when exploring the organization’s data estate.

OKR (Objective-Key-Result) are defined for the business domain and are linked to Data Products for providing context.

One OKR (Objective-Key-Result) was defined as “Decrease the number of seek people in most common diseases”. It was linked to "Sick people by diseases and regions".

One contact was defined to be available to users for asking about this data product.

Critical Data Elements are columns from data assets that are critical pieces of information that are necessary for decision making, and so need to be governed with the highest care. They are defined at business domain level in the Business Domain's Details tab, with data quality purpose. How to create and manage critical data (Preview) | Microsoft Learn. In our Diseases business domain, one example can be the Patient ID.

We’ve associated five data assets in Finance Reports Data Product and one Term, selected from the glossary terms defined in Finance Business Domain:

See that the above data product is of type “Dashboards/Reports”.

You can add as many data assets, terms, OKRs and contacts as needed, depending on your organization needs and size, while the data product is unpublished. Owners can make published data products unpublished to add more data assets, terms, OKRs and critical data elements.

After creation, you can go to Manage Policies and then define several policies for accessing a Data Product.

Come back to “Sick people by diseases and regions” Data Product and press “Manage Policies”.

Among other features, the Preview Access Form in the Policies allows you to define how other users can request access to this Data Product, for example:

You can observe in next figure the list of Data Products we’ve created:

One data product in Finance Business Domain (“Finance Reports”)
Two data products in Diseases Business Domain (“Medicines” and “Sick people by diseases and regions")
“Team and Members” in the Company Projects Business Domain
“Workers” in Human Resources Business Domain

Summarizing concepts

Data Map is the solution where you plan your domains and collections for accepting data assets directly from data sources. Scanning process can be scoped or not depending on the data source.

Domains are managed in the Data Map to define containers for collections and assets.

Business domains are defined and mapped with the desired Collections in Data Catalog.

Business Domains are composed by a set of business concepts, they are:

Data Products,

Glossary terms,

OKRs and

Critical Data.

Data products are composed of related data assets discovered by scanning sources in the data map (you may need to select data assets from more than one source) and curated in the Data Catalog, to satisfy end user's needs.

Glossary Terms and OKRs are defined at business domain level but are linked to Data Products to provide context.

Critical Data are defined for managing data quality as key purpose.

Data Products defined in Diseases Business Domain include several reports, semantic models, data pipelines, and a Lakehouse. These assets reside in a Fabric tenant (the data source) and were selected from the scanned data assets among different Fabric’s workspaces.

We have curated the data products with policies and roles to control their access by users within our organization, as well as with glossary terms to make these assets more readable and easier to find for physicians and other stakeholders.

Learn more:

Understand domains in the Microsoft Purview Data Map | Microsoft Learn

How to manage domains and collections | Microsoft Learn

Microsoft Purview collections architecture and best practices | Microsoft Learn

Permissions in new Microsoft Purview Data Catalog preview | Microsoft Learn

Get ready for the next enhancement in Microsoft Purview governance solutions | Microsoft Learn

Business domains in Microsoft Purview (Preview) | Microsoft Learn

Data products in Microsoft Purview (Preview) | Microsoft Learn

How to configure and manage data catalog access policies (Preview) | Microsoft Learn

Adopting Public IPv6 for Three-Tier Web Applications

jasonmedina — Wed, 14 Aug 2024 16:39:51 GMT

With public IPv4 addresses nearing full allocation, the costs and effort of maintaining IPv4 public IPs for your workloads will only increase. Using IPv6 public addresses can resolve this; there are more of them, and they are more affordable to acquire. Doing so also improves compatibility any IPv6-primary clients, such as IoT devices.

With Application Gateways now supporting dual-stack configuration, you can use an IPv6 address as your front-end for web applications in Azure. Making this change only impacts the front-end; you do not need to assign any internal IPv6 address space to use this, and you can continue to use an IPv4 front-end where needed..

This document streamlines the process of exposing your current web applications to the internet via IPv6 while continuing to run IPv4 on your Azure Virtual Machines. This scenario is ideal for those who need IPv6 exposure but do not require full adoption of IPv6 within Azure.

Existing Solution

The entire environment operates on IPv4 and consists of a single Virtual Network with four subnets:

AppGwSubnet

Contains an Application Gateway that acts as the frontend, load balancing traffic to the web servers in the WebSubnet.

WebSubnet

Contains two IIS web servers that direct traffic to the AppServer Internal Load Balancer VIP in the AppSubnet, which distributes the load among the AppServers.

AppSubnet

Contains two AppServer that direct traffic to the Database Internal Load Balancer VIP in the DataSubnet, which distributes the load among the database servers.

DataSubnet

Contains two database servers using Master/Slave replication that respond to queries from the AppServers.

Step-by-Step Adoption Process

1. Develop an IPv6 address plan and update your virtual network with an IPv6 address space.

1a. Refer to the Conceptual planning for IPv6 networking for guidance on planning your IPv6 networking strategy. For IPv6, it is best practice to deploy a /56 prefix for your Virtual Network and /64 prefixes for your subnets.

Conceptual planning for IPv6 networking - Azure Architecture Center | Microsoft Learn

1b. Add an IPv6 address to your virtual network and to the subnet associated with your Application Gateway to support a dual-stack (IPv4 and IPv6) configuration

Add IPv6 to Virtual Network and Subnet

Note: If your subnet currently hosts an Application Gateway SKU V1, you will need to create a new subnet to deploy a Dual-Stack Application Gateway. However, if you are using Application Gateway SKU V2, you can deploy the Dual-Stack Application Gateway within the same subnet.

2. Deploy a New Dual-Stack Application Gateway and Configure new IPv4 and IPv6 Frontend IPs

2a. Set up a new Application Gateway with dual-stack support to handle both IPv4 and IPv6 traffic. During its creation, assign new public frontend IP addresses for both IPv4 and IPv6.

Create a Dual-Stack Application Gateway

Create IPv4 and IPv6 Frontend IP's

2b. Ensure the new Dual-Stack Application Gateway is configured with the same settings as the original. This includes Listeners with TLS Certificates (for HTTPS/TLS offload), Routing Rules with Backend HTTP Settings (including certificates for End-to-End TLS), Backend Pools, and Health Probes.

2c. Both your IPv4 and IPv6 frontend IPs will use the same Web Application backend pool. Ensure the backend pool is healthy before proceeding to the next step.

3. Update Public DNS Records

3a. Update the DNS ‘A’ record to point to the new dual-stack public IPv4 frontend IP address. Similarly, update the DNS ‘AAAA’ record to point to the new dual-stack public IPv6 frontend IP address.

Create an Azure Public DNS Record

Note: If you are using Public DNS in Azure, please follow the link above. If you are using a different Public DNS service, ensure that the records are updated accordingly

4. Decommission the Original Application Gateway

4a. Once you’ve updated the DNS records and confirmed that your new IPv4 and IPv6 frontend IP addresses are operational on your dual-stack Application Gateway, you can safely delete the original IPv4-only Application Gateway.

Learn More:

Stay Updated on Azure Products Supporting IPv6

https://learn.microsoft.com/azure/architecture/networking/guide/ipv6-ip-planning#configure-azure-services-to-use-ipv6

What is IPv6 for Azure Virtual Network?

https://learn.microsoft.com/azure/virtual-network/ip-services/ipv6-overview

Creating a Local Network Virtual Appliance in Azure for Oracle Database@Azure

adelagarde — Thu, 26 Sep 2024 13:03:44 GMT

Oracle Database@Azure is an Oracle database service running on Oracle Cloud Infrastructure (OCI), collocated in Microsoft data centers. This ensures that the Oracle Database@Azure service has the fastest possible access to Azure resources and applications. The solution is intended to support the migration of Oracle database workloads to Azure, where customers can integrate and innovate with the breadth of Microsoft Cloud services. For more information and to gain a better understanding of Oracle Database@Azure please visit Overview - Oracle Database@Azure | Microsoft Learn

The current Oracle Database@Azure service has a network limitation where it cannot respond to network connections outside of its Azure virtual network (VNet ) when it is expected to route through a firewall. This limitation places constraints on extending integration to Azure services not located within the same Vnet. This issue also impacts network communication from on-premises that need to connect to the Oracle Database@Azure service.

To address this network limitation, the recommended solution is to deploy a Network Virtual Appliance (NVA) within the Oracle Database@Azure VNet. While Microsoft and Oracle are working together to develop an update to the Azure platform that will eliminate this limitation, customers will need to follow this design pattern until the official rollout of the update.

Deploying an NVA

The NVA consists of a Linux virtual machine (VM) and any supported distribution on Azure can be used. The NVA referenced in this article is not a traditional firewall, but a VM acting as a router with IP forwarding enabled and not intended to be an enterprise-scale Firewall NVA. This solution is only expected to help customers bridge the gap until the jointly engineered design pattern is available in all Azure regions.

The deployment of the NVA helps solve the specific scenarios outlined below:

Where traffic inspection is required between Oracle Database@Azure and other resources
Where native network support is not available
With resources that have private endpoints
Resources on another Azure virtual network (VNet)
Services with delegated subnets
Connectivity with on-premises

Additional details on supported network topologies can be found within the following article Network planning for Oracle Database@Azure | Microsoft Learn

Scope

This article's scope will review a network scenario within an Azure Landing Zone requiring an NVA. The deployment steps of the NVA and other ancillary steps required to complete the end-to-end implementation are included. This article does not cover the hybrid connectivity from on-premises to Azure. That scenario will be covered in a later article; however, both share the same method of using User Defined Routes (UDR).

Scenario Review

The Azure Landing Zone consists of Hub and Spoke architecture where the application layer is hosted in a Vnet specific for the application front end services, such as web servers. The Oracle Database@Azure is deployed in a separate dedicated Vnet for data. The goal is to provide bidirectional network connectivity between the application layer and the data layer.

Deployment

The steps provided in this article should be followed in the designated order to ensure the expected results. Please consult with either your Microsoft or Oracle representative if you have specific questions related to your environment.

Environment Overview

Hub VNet (10.0.0.0/16)

Hub NVA: 10.0.0.4

Spoke 1 VNet - Application Tier (10.1.0.0/16)

Application Server: 10.1.0.4

Spoke 2 VNet - Oracle Database (10.2.0.0/16)

Oracle DB Subnet: 10.2.0.0/24
Oracle Database: 10.2.0.4
Local NVA Subnet: 10.2.1.0/24
Local NVA: 10.2.1.4

Note: At the time this article was published, Azure Firewall is currently not supported in this scenario. Third-party NVA’s native support is scheduled for 2024, but subject to change. Third-Party NVA's require this workaround to satisfy the above-mentioned scenario to support network communication until these features are fully implemented on Azure.

Create a Linux VM in Azure as an NVA

Set up a Linux VM (using any supported distributions on Azure) in the desired resource group and region as the Oracle Database@Azure using your deployment method of choice (for example Azure portal, Azure PowerShell, or Azure CLI). As a security recommendation, be sure to leverage Secure Shell (SSH) public/private keys to ensure secure communication.

Ensure the VM is in the same Vnet, but on a separate subnet from the Oracle Database@Azure delegated subnet as well as the dedicated Oracle backup subnet if it has been deployed

Note: Sizing is very much driven by the actual traffic pattern. Consider how much traffic (volume) packets per second are required to support the implementation. Starting with a 2-core general-purpose VM (D2s_v5 with 2 vCPUs) and 8 GiB (gibibytes) of memory including accelerated networking which can be used to gauge initial performance. High storage/IOPS performance SKUs are not necessary for this use case.

As part of the deployment and monitoring strategy please consult Welcome | Azure Monitor Baseline Alerts for the proper Azure Monitor counters that should be enabled against the NVA to ensure performance and availability.

Enable IP Forwarding on the VM's NIC (Network Interface Cards)

Go to the Networking section of the NVA VM in the Azure portal
Select the Network Interface
Under Settings, choose IP configurations
Enable IP forwarding

Enable IP Forwarding at the Operating System level

SSH into the VM.
Edit the sysctl configuration file to enable IP forwarding: sudo nano /etc/sysctl.conf
Uncomment the following line: net.ipv4.ip_forward = 1
Save and exit out of nano to apply the changes
Run the following command to reset the network status to forward network traffic without a reboot on the VM: sudo sysctl -p and hit enter. You should see the following line net.ipv4.ip_forward = 1 will appear on the screen indicating the changes were made successfully.

We now need to implement iptables rules to route traffic properly through the NVA. When using iptables after a reboot Linux systems will lose their iptables rules. In order to avoid that we will install some packages and make some configurations. In the first example we will use either an Ubuntu or Debian Linux distribution. We will only be using IPv4 with the following changes on the Linux system listed in this article.

Ubuntu / Debian Linux system

Ensure that the local firewall on the NVA is enabled or set to not block traffic. First start by enabling iptables by running the following command sudo systemctl enable iptables and hit enter. Then type sudo systemctl start iptables and hit enter.

To list the current iptables rules by running the following command sudo iptables -L and hit enter. This will list any possible firewall rules.

Note: If there are rules disable them with the following command sudo iptables -F and hit enter.

We need to install a package called iptables-persistent by typing the following command:

On a Ubuntu system type sudo apt install iptables-persistent and hit enter.

On a Debian system type sudo apt-get install iptables-persistent and hit enter.

Make sure services are enabled on Debian or Ubuntu using the systemctl command:

sudo systemctl is-enabled netfilter-persistent.service hit enter.

If not enabled type the following command:

sudo systemctl enable netfilter-persistent.service hit enter.

Get the status of the service by running the following command:

sudo systemctl status netfilter-persistent.service hit enter

Enter the following commands line by line and hit enter for each:

sudo iptables -t nat -A POSTROUTING -j MASQUERADE

sudo iptables -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT

sudo iptables -A FORWARD -j ACCEPT

Validate the iptables rules are in place by typing sudo iptables -L and hit enter.

The iptables rules applied will be saved and loaded if the system reboots.

In the second example we will use either a RedHat Enterprise Linux System (RHEL), Fedora, and AlmaLinux. The system commands are similar for the following Linux distributions.

RHEL/Fedora/AlmaLinux

Type the following commands line by line and hit enter for each to disable firewalld:

sudo systemctl stop firewalld.service
sudo systemctl disable firewalld.service
sudo systemctl mask firewalld.service

Next, we must install the iptables-services package by either using the native yum or dnf package management commands.

The following example uses yum. Type each the following command line by line followed by hitting enter for each:

sudo yum install iptables-services
sudo systemctl enable iptables
sudo systemctl enable ip6tables
sudo systemctl status iptables

If we use dnf enter each line by line and hit enter :

sudo dnf install iptables-services
sudo systemctl enable iptables
sudo systemctl enable ip6tables
sudo systemctl status iptables

Once the service is installed, you can configure the /etc/sysconfig/iptables file for IPv4. Any rules added to this file makes them persistent. You can use your favorite editor vi, vim, or nano to edit the file. Add the following line by line and save the file once complete.

sudo iptables -t nat -A POSTROUTING -j MASQUERADE

sudo iptables -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT

sudo iptables -A FORWARD -j ACCEPT

Next, we need to load the changes that were just made by typing the following command:

sudo systemctl restart iptables and then hit enter.

Ensure that the Network Security Group (NSG) on the NVA is allowing all traffic from the application Vnet and the Oracle Database@Azure delegated subnet.

Configure Route Tables

Oracle Database@Azure Vnet (Spoke)

Create a route table in the Azure portal in the same region and proper resource group (RG) where the Oracle Database@Azure is located. Give it a meaningful name.
Add routes to the route table:
Oracle Database Subnet: Associate the route table with this subnet.
From Oracle Database Subnet: Set the next hop for 0.0.0.0/0 to the local NVA VM.

Important: Ensure in the configuration of the route table that all route propagation is disabled. This setup ensures that all traffic to and from the Oracle Database is forced through your local NVA.

Configure Route Tables for the NVA in the Oracle Database azure Vnet

Create another route table in the Azure portal in the same region and proper resource group (RG) where the Oracle Database@Azure is located. Give it a meaningful name.
Add routes to the route table:
NVA Subnet: Associate the route table with this subnet.
From NVA Subnet: Set the next hop for 0.0.0.0/0 to the HUB NVA (10.0.0.4).

Route Configuration Application Tier

Route to Hub NVA

Create another route table in the Azure portal in the same region and proper resource group (RG) where the Oracle Database@Azure is located. Give it a meaningful name.
Application Subnet: Attach the route table to the Application Subnet in the application Vnet.
Route from Application Vnet: Destination: 10.2.0.0/24 (Oracle Database Subnet) Next Hop: 10.0.0.4 (Hub NVA)

Route Configuration Hub VNet

Route to Local NVA:

Create another route table in the Azure portal in the same region and proper resource group (RG) where the Oracle Database@Azure is located. Give it a meaningful name.
Firewall Subnet: Attach the route table to the Firewall Subnet in the Hub Vnet.
From Firewall Subnet: Set the next hop 10.2.0.0/24 (Oracle Subnet) to 10.2.1.4 (Local NVA)
Please ensure if you have a Cisco or Palo Alto or other third-party NVA’s that there are no internal static routes that may conflict with the custom route table from Azure.

When finished the implementation network flow and environment should match the following diagram:

Testing

The next step is to start testing by initiating a connection from the application servers. Make sure the proper components have been installed on the application servers to connect to the Oracle Database@Azure before validating connectivity. Validate that the application servers can connect to the Oracle Database@Azure.

If you need to troubleshoot deploy a test Linux vm on the application subnet to test connectivity. Install the mtr package as a tool on the Linux test vm. Do not rely on ping (ICMP) to troubleshoot as this will not properly test connectivity within Azure. An example of the command using mtr would be the following: sudo mtr -T -n -P 1521 10.2.0.4. This example starts a trace attempting to connect to the database without using ICMP. The network port of 1521 is selected which the database listens on for connections. Review the route tables and IP addresses were entered correctly if a problem is identified. If the initial tests are successful, you have implemented this solution correctly.

Next Steps

Please visit the Microsoft Cloud Adoption Framework (CAF ) Introduction to Oracle on Azure adoption scenarios - Cloud Adoption Framework | Microsoft Learn

Authors
Moises Gomez Cortez

Technical Editor and Content Contributor
Anthony de Lagarde, Erik Munson

Securing Microsoft Fabric: User Authentication & Authorization Guidelines

inbalsilis — Mon, 19 Aug 2024 11:50:18 GMT

Did you wonder what are the options to define users and permissions to access and operate in Microsoft Fabric?

Considering Conditional Access for Fabric users?

Looking to understand the best practices to define user roles in workspace level?

In this blog, we will talk about authentication and authorization options in Fabric including use case example.

Microsoft Fabric is a software as a service (SaaS) platform that lets users get, create, share, and visualize data.

Security is a top priority for any organization that wants to succeed in the digital age. You need to safeguard your assets from threats and follow your organization's security policies.

One of the security design principles is Implement strong access controls that authenticate and authorize access to the system.

This blog describes recommendations for authenticating and authorizing users that are attempting to access Microsoft Fabric.

First, let understand Microsoft Fabric main infrastructure components with this diagram:

In this diagram, we can see the different components in Fabric and the possible relation:

Tenant - A tenant is a single instance of Fabric for an organization and is aligned with a Microsoft Entra ID.

OneLake - a single, unified, logical data lake for your whole organization. A data Lake processes large volumes of data from various sources. OneLake is the foundation of Microsoft Fabric for your tenant. There is one OneLake per Tenant.

Capacity - Capacity is a dedicated set of resources that is available at a given time to be used. Capacity defines the ability of a resource to perform an activity or to produce output. Different items consume different capacities at a certain time. Fabric offers capacity through the Fabric SKU and Trials. You can have multiple capacities for one OneLake.

Workspace - A workspace is a collection of items that bring together different functionality in a single environment designed for collaboration. It acts as a container that uses capacity for the work that is executed, and provides controls for who can access the items in it. For example, in a workspace, users create reports, notebooks, semantic models, etc. You can have multiple workspaces at one capacity.

Now, let's examine Entra ID authentication and authorization on top of the layers.

Authentication

Microsoft Entra tenant provides identity and access management (IAM) capabilities to applications and resources used by your organization. Since Fabric is deployed to a Microsoft Entra tenant, authentication and authorization are handled by Microsoft Entra.

Access token authentication:

Fabric relies on Microsoft Entra ID to authenticate users or service principals, when authenticated, users or service principals receive access tokens from Microsoft Entra ID. Fabric uses these tokens to perform operations in the context of the user or application.

Example: When a user logs into Fabric, they are authenticated by Microsoft Entra ID and receive an access token. This token is then used to access various resources within Fabric.

Note:An identity is a directory object authenticated and authorized for access to a resource. There are identity objects for human (users) and nonhuman identities. Human identities are referred to as identities, and nonhuman identities are workload identities. Nonhuman entities include application objects, Service Principals, managed identities, and devices. Generally, workload identity is for a software entity to authenticate with a system. This blog describes authentication options for users/humans while service principal/nonhuman is out of scope.

Non-Access Token authentication:

Non-access token activity in Fabric enables you to utilize external data sharing. Fabric external data sharing is a feature that allows Fabric users to share data from their tenant with users in another Fabric tenant.

Example: If you enable external data sharing, you are explicitly trusting other tenants, allowing them to access the shared data without complying with your Entra Conditional Access Policy. To enforce CA Policy for all cases, it is recommended to turn off external data sharing at the tenant level unless there is a specific need to use such external data.

Secured authentication via Conditional Access (CA)

A key feature of Microsoft Entra ID is conditional access. Conditional access ensures that customers can secure apps in their tenants, including:

Multifactor authentication
Allowing only Intune enrolled devices to access specific services
Restricting user locations and IP ranges

To configure conditional access for users in Fabric, you need to select several Azure services: Power BI, Azure Data Explorer, Azure SQL Database, and Azure Storage.

Setup of Power BI, Azure Data Explorer, Azure SQL Database, and Azure Storage block any access token activity in Fabric incase CA policy fails.

Authorization

All Fabric permissions are stored centrally by the metadata platform. Fabric services query the metadata platform on demand in order to retrieve authorization information and to authorize and validate user requests.

In this blog, we will focus on Workspace level permissions, you can read more on Item permissions here

Authorization in Workspace level

Organizational teams can have individual workspaces where different personas collaborate and work on generating content. Access to the items in the workspace is regulated via workspace roles assigned to users by the workspace admin.

You can either assign roles to individuals or to groups.

Guidance: Data owners should recommend users who could be workspace administrators. These could be team leaders in your organization, for example. These workspace administrators should then govern access to the items in their workspace by assigning appropriate workspace roles to users and consumers of the items.

There are four Workspace roles and they apply to all items within the workspace. Users that don't have any of these roles can't access the workspace. The roles are:

Viewer - Can view all content in the workspace but can't modify it.
Contributor - Can view and modify all content in the workspace.
Member - Can view, modify, and share all content in the workspace.
Admin - Can view, modify, share, and manage all content in the workspace, including managing permissions.

You can define workspace level access in Fabric via:

UI as explained in Give users access to workspaces via UI

API as explained in Add Workspace Role Assignment via Fabric API

This diagram demonstrates the Authentication and Authorization options described above:

What can you do to manage access easily and efficiently?

Microsoft Entra groups are used to manage users that all need the same access and permissions to resources, such as potentially restricted apps and services. Instead of adding special permissions to individual users, you create a group that applies the special permissions to every member of that group.

When user leave the organization, you can easily remove the user form the group, that will remove user access to the different workspaces.

Let look on one example: Organization with multiple users granting access to Fabric dev, test and prod workspaces via Entra groups.

The first step will be to define groups for Fabric users in Microsoft 365 admin center.

Potential groups implementation:

1.Fabric all - include all Fabric users that will get access to Fabric

2.Development + Test users:

Fabric dev+test workspace viewers - assign users per need
Fabric dev+test workspace members - assign users per need
Fabric dev+test workspace contributors - assign users per need
Fabric dev+test workspace admins - potentially assign to workspace owners + team leads

3.Production users:

Fabric prod workspace viewers - assign users per need
Fabric prod workspace members - assign users per need
Fabric prod workspace contributors - assign users per need
Fabric prod workspace admins - potentially assign to workspace owners + team leads

Here you can see sample groups from Microsoft 365 admin center:

The next steps will be:

Define Conditional Access policy for Fabric users, via the group we created before - Fabric all, as explained in Conditional access - Microsoft Fabric | Microsoft Learn

Define workspace level access in Fabric UI as explained in Give users access to workspaces or via API as explained in Workspaces - Add Workspace Role Assignment

Here we can see sample UI of assigning Admin rights on the workspace via the group we created before:

1.Go to Manage Access

2.Add the relevant group:

3.Assign permission to the group:

In conclusion, implementing strong access controls to authenticate and authorize users is a crucial security design principle for any organization using Microsoft Fabric. Understanding the different components and layers of the platform can aid in the configuration of authentication and authorization options, such as Entra ID and Conditional Access policies.

Conditional Access provides an additional layer of security and requires Microsoft Entra ID P1 licenses.
Authorization at the workspace level is regulated via workspace roles, which can be assigned to users or groups.
It is recommended that data owners assign workspace administrators and govern access to the workspace by assigning appropriate roles to users and consumers of the items

Additional reference: End-to-end security overview for Fabric can be found here

Assaf & Inbal.

Recover Multiple VMs from Azure Backup in Less Time

sdolgin — Sat, 03 Aug 2024 11:25:10 GMT

In the dynamic world of cloud computing, time is often a critical factor, especially when it comes to recovering from disasters like ransomware attacks or rolling back after a problematic security update. Imagine waking up to find your entire set of Azure VMs compromised by ransomware or discovering that a recent security update has left your systems inoperable. The clock starts ticking, and the longer it takes to restore your VMs, the greater the impact on your business.

The Problem

Azure Backup is a robust solution for protecting your Azure VMs, providing peace of mind with its ability to create and manage backup policies, configure backup schedules, and perform reliable restores. However, the features available for Azure Backup in the Portal UI today only allow you to start the restoration of individual VMs one at a time in a sequence of repeated steps. Restoring many VMs manually through the Azure Portal can be extremely time-consuming and inefficient, especially when you need to restore an entire set of VMs quickly.

Native Capabilities of Azure Backup

Azure Backup offers extensive features for protecting your VMs:

Automated backup schedules and retention policies - A backup policy can protect multiple Azure VMs consisting of a schedule (daily/weekly) and retention (daily, weekly, monthly, yearly).
Cross-region restore capabilities – Allows you to restore data in a secondary, Azure paired region. This can be useful to conduct BCDR drills or if there’s a disaster in the primary region.
Ransomware protection – Features such as irreversible soft-delete, immutable storage and multi-user authorization can be set at the vault level to safeguard backup data.

Despite these powerful features, there is always room for improvement. For now, there is currently no feature in the Azure Portal to start a batch restore multiple VMs simultaneously. This limitation becomes a bottleneck in scenarios where speed and efficiency are paramount.

Az PowerShell Module – A Good Solution

To address this gap, we turn to PowerShell scripting, a versatile and powerful tool for managing Azure resources. Microsoft's Az PowerShell module provides a comprehensive suite of cmdlets to automate and manage Azure tasks, including VM backups and restores.

Here's a practical approach: a PowerShell script that enables the parallel restoration of multiple VMs from Azure Backup. This script leverages Az.RecoveryServices to streamline the recovery process, significantly reducing the time required to get your systems back online.

Sample PowerShell Script

Below is a summary of how one example script works. You can find the full example here.

Define Variables:
The script starts by defining the necessary variables, including the resource group, recovery services vault, and cache storage account.
$resourceGroup = "rg-webservers" $recoveryServicesVault = "rsv-vmbackup" $cacheStorageAccount = "unique626872" $cacheStorageAccountResourceGroup = "rg-webservers"
Get the Vault and Backup Container:
It then retrieves the Recovery Services Vault and Backup Container containing multiple VMs that need to be restored in parallel.
$vault = Get-AzRecoveryServicesVault -ResourceGroupName $resourceGroup -Name $recoveryServicesVault $container = Get-AzRecoveryServicesBackupContainer -ContainerType "AzureVM" -VaultId $vault.ID
Loop Through the Backup Items:
The script iterates over each backup item in the container, performing a shutdown and kicking off an in-place restore from the latest recovery points.
foreach ($item in $container) { # Write out the Backup Item Name Write-Host "Backup Item Name: $($item.FriendlyName)" # Get the Backup Item from the Vault $backupItem = Get-AzRecoveryServicesBackupItem -BackupManagementType "AzureVM" -WorkloadType "AzureVM" -VaultId $vault.ID -Name $item.Name # Get the resource group & VM name for this backupItem $vmResourceGroup = $backupItem.VirtualMachineId.Split('/')[4] $vmName = $backupItem.VirtualMachineId.Split('/')[8] # Shut down the protected VM before restoring it Write-Host "Stopping VM: $vmName in Resource Group: $vmResourceGroup before restoring" Stop-AzVM -ResourceGroupName $vmResourceGroup -Name $vmName -Force -SkipShutdown -NoWait # Get the latest recovery point for the backup item from the last 7 days $recoveryPoints = Get-AzRecoveryServicesBackupRecoveryPoint -Item $backupItem -VaultId $vault.ID -StartDate (Get-Date).AddDays(-7).ToUniversalTime() -EndDate (Get-Date).ToUniversalTime() # Write details about the latest recovery point Write-Host "Found $($recoveryPoints.Count) Recovery Points for the Backup Item" Write-Host "Latest Recovery Point Time: $($recoveryPoints[0].RecoveryPointTime)" # Extract necessary properties from the recovery point $recoveryPointId = $recoveryPoints[0].RecoveryPointId # Restore the Azure VM from the latest recovery point to the original location (replace the source VM) # To speed up the process we run the VM restores in parallel using PowerShell jobs Write-Host "Restoring $($item.FriendlyName) to the original location" $job = Start-Job -ScriptBlock { param ($recoveryPointId, $backupItemName, $vaultId, $vaultLocation, $cacheStorageAccount, $cacheStorageAccountResourceGroup) # Retrieve the backup item and recovery point within the job $vault = Get-AzRecoveryServicesVault -ResourceGroupName $vaultId.ResourceGroupName -Name $vaultId.Name $backupItem = Get-AzRecoveryServicesBackupItem -BackupManagementType "AzureVM" -WorkloadType "AzureVM" -VaultId $vault.ID -Name $backupItemName $recoveryPoint = Get-AzRecoveryServicesBackupRecoveryPoint -Item $backupItem -VaultId $vault.ID | Where-Object { $_.RecoveryPointId -eq $recoveryPointId } Restore-AzRecoveryServicesBackupItem -RecoveryPoint $recoveryPoint -StorageAccountName $cacheStorageAccount -StorageAccountResourceGroupName $cacheStorageAccountResourceGroup -VaultId $vault.ID -VaultLocation $vault.Location } -ArgumentList $recoveryPointId, $item.Name, $vault, $vault.Location, $cacheStorageAccount, $cacheStorageAccountResourceGroup # Store the job information $jobs += $job # Write out the Job ID Write-Host "Started Restore Job ID: $($job.Id)" }

By using this script, you can dramatically reduce the time required to restore multiple VMs, enhancing your ability to recover from critical incidents swiftly.

Conclusion

In this post, we've discussed the limitations in the Azure Portal for handling multiple VM restores and introduced a practical workaround using PowerShell scripting. This solution enables you to restore your VMs in parallel, significantly cutting down recovery time and minimizing business disruption.

We encourage you to modify the sample script, try it out in your test environment and see the benefits for yourself. Your feedback is invaluable, so please share your experiences and let us know your thoughts on this approach. Together, we can continue to improve and innovate in the realm of Azure Business Continuity.

Call to Action

Modify the sample script and try it out in your test environment for parallel VM restoration.
Share your feedback and experiences with us.
Stay tuned for more tips and tricks on maximizing your Azure capabilities.

Happy scripting, and may your recoveries be swift and seamless!

FastTrack for Azure articles

Modernizing On‑Prem File Servers: Azure Storage Mover and File Sync

Azure Firewall and Service Endpoints

The Design

Network Rules or Application Rules?

Conclusion

Azure Private Endpoint vs. Service Endpoint: A Comprehensive Guide

FastTrack for Azure (FTA) program retiring December 2024

Azure Backup vs. Azure Site Recovery: Key Differences Explained

Oracle Database@Azure DNS Setup

Introduction

Scenario

DNS Deployment options for Oracle Database@Azure

DNS prerequisites before you deploy Oracle Database@Azure in Azure

Oracle Database@Azure default DNS setup workflow

Oracle Database@Azure custom DNS setup workflow

Oracle Database@Azure external DNS setup workflow

Custom DNS

Next Steps

Connecting to Azure SQL Database using SQLAlchemy and Microsoft Entra authentication

Comparing Azure SQL Authentication Methods

SQL Authentication Drawbacks

Improved Security with Microsoft Entra authentication

Pre-requisites

Configure the Database

Setting Current User as Azure SQL DB Admin

Adding Your IP Address to the Azure SQL Server Firewall

Set up the project

Running Locally

Running on Azure

References

Securing Your Data Pipelines: Best Practices for Fabric Data Factory

Key Security Features in Fabric Data Factory

Example: Securing a Data Pipeline in Fabric Data Factory

Scenario:

Prerequisites:- Tools and Technologies Needed:

Steps:Step 1: Enable Managed Identity in Workspace level for Fabric Data Factory pipeline

Step 2: Configure trusted workspace access in ADLS Gen2

Step 3: Create a pipeline to connect to ADLS gen2 and copy data to Fabric Lakehouse

Step 4: Results

After copy activity finish running, you can view the data in your Lakehouse

Conclusion

Making Searching and Curating Data Assets in Microsoft Purview easier.

Implementing Route Summarization in Azure VMware Solution

What is Route Summarization?

Why Use Route Summarization in Azure VMware Solution?

Route Summarization in NSX

Scenario Overview

Before configuring Route Summarization

Entra ID federation with Google Workspace

Requirements

Configuring SAML Federation on Google Workspace side

Adding Federation for SAML Provider in Entra ID

How to monitor applications by using OpenTelemetry on Azure Container Apps

Overview

How to Monitor Applications on Azure Container Apps

Demo Application Specifications

1. Using Application Insights (Azure Monitor OpenTelemetry Distro)

Transaction view

Metric

2. Using the OpenTelemetry Collector of the Container Apps Environment

How to Enable OpenTelemetry collector

Transaction view

3. Using Dapr

How to enable Dapr based instrumentation

Transaction view

4. Forwarding System Logs/Console Logs to Log Analytics

How to view logs

Monitoring Applications with Application Insights and OpenTelemetry

What Application Insights Can Do

What OpenTelemetry Can Do

Conclusion

Leveraging dynamic few-shot prompt with Azure OpenAI

Microsoft Fabric Metadata Driven Pipelines with Mirrored Databases

Metadata Driven Pipelines with Microsoft Fabric Mirrored Databases

Architecture Overview

Why 2 semantic models? Fabric Data Warehouse vs Mirrored Database SQL Endpoint for semantic model reporting

Solution details

Migrating from Azure APIM STv1 to STv2: New Options and Considerations

Explaining Purview concepts: Domains, Business Domains, Collections, Data Products and Data Assets.

Prerequisites:
- Tools and Technologies Needed:

Steps:

Step 1: Enable Managed Identity in Workspace level for Fabric Data Factory pipeline