Part 7 of a 9-part series . Today’s post is the first of two sections; to read the second half, click here .
One of the foundational requirements we called out in the 2012 R2 vision document was our promise to help you transform the datacenter. A core part of delivering on that promise is enabling Hybrid IT.
By focusing on Hybrid IT we were specifically calling out the fact that almost every customer we interacted with during our planning process believed that in the future they would be using capacity from multiple clouds. That may take the form of multiple private clouds an organization had stood up, or utilizing cloud capacity from a service provider or a public cloud like Azure, or using SaaS solutions running from the public cloud.
We assumed Hybrid IT would really be the norm going forward, so we challenged ourselves to really understand and simplify the challenges associated with configuring and operating in a multi-cloud environment. Certainly one of the biggest challenges associated with operating in a hybrid cloud environment is associated with the network – everything from setting up the secure connection between clouds, to ensuring you could use your IP addresses (BYOIP) in the hosted and public clouds you chose to use.
The setup, configuration and operation of a hybrid IT environment is, by its very nature incredibly complex – and we have poured hundreds of thousands of hours into the development of R2 to solve this industry-wide problem.
With the R2 wave of products – specifically Windows Server 2012 R2 and System Center 2012 R2 – enterprises can now benefit from the highly-available and secure connection that enables the friction-free movement of VMs across those clouds. If you want or need to move a VM or application between clouds, the transition is seamless and the data is secure while it moves.
The functionality and scalability of our support for hybrid IT deployments has not been easy to build, and each feature has been methodically tested and refined in our own datacenters. For example, consider that within Azure there are over 50,000 network changes every day, and every single one of them is fully automated. If even 1/10 of 1% of those changes had to be done manually, it would require a small army of people working constantly to implement and then troubleshoot the human errors. With R2, the success of processes like these, and our learnings from Azure, come in the box.
Whether you’re a service provider or working in the IT department of an enterprise (which, in a sense, is like being a service provider to your company’s workforce), these hybrid networking features are going to remove a wide range of manual tasks, and allow you to focus on scaling, expanding and improving your infrastructure.
In this post, Vijay Tewari (Principle Program Manager for Windows Server & System Center) and Bala Rajagopalan (Principle Program Manager for Windows Server & System Center), provide a detailed overview of 2012 R2’s hybrid networking features, as well as solutions for common scenarios like enabling customers to create extended networks spanning clouds, and enabling access to virtualized networks.
Don’t forget to take a look at the “ Next Steps ” section at the bottom of this post, and check back tomorrow for the second half of this week’s hybrid IT content which will examine the topic of Disaster Recovery.
* * *
Hybrid networking refers to capabilities that seamlessly extend an enterprise’s on-premises network to the service provider’s (hosted or Azure) cloud, and many teams across Microsoft have come together to deliver these end-to-end experiences and scenarios for our customers.
Hybrid networking enables enterprises to easily move their VMs (and virtualized workloads) across clouds while maintaining IP addresses and other networking policies. With hybrid networking, an enterprise administrator can treat a composite network spanning on-premises cloud boundaries as a single extended network .
As enterprises continue to utilize more and more capacity from the cloud, service providers face a pressing issue: How does one simplify the to-the-cloud migration of an increasing number of tenants while minimizing their capital and operational expenses?
Hybrid networking impacts both the ease of workload migration as well as the cost to the service provider. To maximize this benefit, Windows Server 2012 R2 and Systems Center 2012 R2 have been built to deliver cloud-optimized server and management capabilities that enable service providers to deliver efficient hybrid networks that easily onboard tenants.
These capabilities broadly fall under three specific areas we’ll examine today:
- Cloud connectivity
- Network virtualization
- Network infrastructure management
This post will also cover 3 key scenarios that have been designed into the 2012 R2 wave of products.
C loud connectivity
Windows Server 2012 included the cross-premises gateway feature, which allowed enterprise sites to be connected to each other or to the cloud using standard VPN protocols. In Windows Server 2012 R2, this feature is extended into a full-fledged site-to-site (S2S) VPN gateway to support cloud service provider and enterprise customer requirements (see Figure 1 below).
These customer requirements include:
-
Enhanced S2S VPN
This retains industry standard IPsec/IKEv2 tunneling technology as before, but with the additional capabilities to rate-limit traffic in both directions, and use tenant-specific traffic filters. -
Multitenancy
A single gateway (deployed as a VM) can support S2S connections from multiple customers, thus eliminating the need to deploy separate gateways for each customer and saving costs for the service provider. -
High availability
S2S VPN gateway can be deployed in 1+1 configuration for high availability (HA). -
Multi-site topologies
and
automatic routing
Multiple sites of a customer can be connected to the service provider cloud over S2S connections with automatic determination of destinations reachable over different connections using the industry standard Border Gateway Protocol (BGP). -
Integration with Hyper-V Network Virtualization for tenant network isolation
Hyper Network Virtualization (HNV) enables the service provider and enterprise to deploy virtualized customer networks with overlapping IP addresses over the same physical network infrastructure with complete traffic isolation. The S2S VPN gateway natively interfaces with HNV to bridge traffic between the customer’s virtualized network provisioned in the cloud with their on-premises network. -
Integrated Network Address Translator (NAT)
Applications running in the customer’s virtualized network can directly access public sites in the Internet using the built-in NAT functionality rather than reaching the Internet through the S2S connection and the customer’s on-premises network. -
VPN access
Users from the customer’s organization who are off-premises can access VMs and services in the customer’s virtualized network in the cloud using the built-in multitenant VPN functionality. -
Integration with Systems Center Virtual Machine Manager
(
Provisioning, configuration, and management of the gateway can be done using System Center 2012 R2, Virtual Machine Manager (VMM), VMM 2012 R2 , and a Windows Azure Pack -based self-service portal can be offered by the service provider for customers to create and monitor S2S connections.
Figure 1
Network virtualization
With the 2012 R2 release, we have further refined the Hyper-V Extensible Switch and Hyper-V Network Virtualization which were both introduced in Windows Server 2012.
The Hyper-V Extensible Switch now allows third party extensions to act on packets before and after HNV encapsulation for a richer set of policy and filtering implementations. Additionally, support for hybrid forwarding is available – this allows different forwarding agents to handle packets based on type, e.g. HNV or non-HNV encapsulated.
The impact of this is big: Multiple network virtualization solutions (one provided by HNV, and another from the forwarding switch extension) can co-exist on the same Hyper-V host. Regardless of which agent performs the forwarding computation, the forwarding extension can apply additional policies to the packet. More details on the 2012 R2 Hyper-V Extensible Switch features can be found here .
HNV also now supports dynamic IP address learning, an in-box S2S gateway (covered in this article), integration with NIC teaming for load balancing, high availability, and NVGRE task offload support from NIC vendors. More info on these elements can be found in this blog article .
In addition to the features noted above, Virtual Receive-Side Scaling (vRSS) in the R2 release enables traffic-intensive workloads to scale up to optimally utilize high speed NICs (10G) even from a VM. In addition to this, in an effort to improve diagnostics and thereby drive down opex, we have enabled remote packet captures and ETW traces in near real-time, without a need to log on to the destination machine.
We have also enabled basic management of physical switches (with standards-based schemas) using PowerShell and VMM, thereby making it possible to automate certain diagnostics across the hosts and into the physical network. Finally, deployment and management are now entirely automated via VMM, with tenant self service via the Windows Azure Pack (as shown in Figure 1).
For a deeper dive on the network virtualization enhancements in R2, check out
this recent post
on the
Networking Blog
.
Network infrastructure management
Windows Server 2012 IP Address Management (IPAM) solution supported core IP address, DHCP and DNS management functions. With the R2 release, IPAM has implemented several major enhancement, such as:
- Unified IP address space management of physical and virtual networks through tight integration with VMM.
- Granular and customizable role-based access control and delegated administration across multiple datacenters.
- Single console monitoring and management of DHCP and DNS services across datacenters – in particular, the administration of DHCP failover, DHCP policies, and filters.
- Complete PowerShell support for integration with automation workflows.
- Support for SQL Server as IPAM data store.
There are also enhancements to the addressing and naming services themselves, including per-zone query metrics support in DNS, and FQDN-based DHCP policies. The use of IPAM in a hoster datacenter is depicted in Figure 1.
Our recent TechNet article on IPAM in Windows Server 2012 R2 describes these enhancements in more detail.
Now, to illustrate how these technologies have been built and bundled together to serve as solutions for common scenarios encountered by IT pros and service providers, the following three scenarios show how hybrid networking can make an infrastructure environment more powerful, scalable, resilient, and manageable.
Scenario 1: Enabling an enterprise to create an extended network spanning their on-premises site and the service provider cloud
We described the integrated approach we have taken in enabling our customers to deploy an IaaS solution in an earlier post in this R2 series , and as service providers and enterprises deploy IaaS, their customers in turn deploy their VM’s onto this capacity.
Traditionally these VM’s were provided IP addresses from a service provider network address space thereby forcing customers to renumber their VMs and rethink their network environments. With the availability of network virtualization, customers can create their own virtualized network on top of the service provider network infrastructure, utilize their own private IP numbering plan within the virtualized network, and connect it to their on-premises network to create an extended network that spans the on-prem infrastructure and the service provider cloud.
This approach dramatically simplifies the migration of workloads to the cloud and back.
This scenario is enabled by a combination of two key aspects: First, network virtualization, which enables the service provider to run multiple tenant networks with overlapping IP addresses on the same physical infrastructure. Second, the capability to connect the customer’s virtualized network in the service provider cloud back to the customer’s on-premises network to form one extended network.
To enable this deployment, the service provider provisions the network infrastructure and a set of S2S VPN gateways to support the creation of customer (virtualized) networks and S2S connections. Next, the customer self-provisions his virtualized network and S2S connections using the Windows Azure pack portal.
Now, let’s look at each step of this scenario in detail:
Configure the infrastructure for network virtualization and deploy gateway VMs.
- The service provider admin creates a fabric network using VMM that allows a customer to deploy their virtual networks.
- After this, the admin deploys a set of VMs either manually or using VMM service templates .
- He then adds each VM as a S2S gateway.
- These steps are detailed in this guide .
Deploy Windows Azure Pack
- Deploying the Windows Azure Pack allows customers to self-provision their virtual networks.
- Details on deploying Windows Azure Pack can be found here .
Customer self-provisioning of virtualized network and S2S connections
- The customer logs into the self-service portal of the service provider (enabled by deployment of Windows Azure Pack) and creates his virtualized network in the provider’s cloud.
- Now the customer is able to choose his own IP numbering plan for the virtualized network, which is typically from the customer’s private address space.
- Following this, he provisions S2S connections by specifying the address of the gateway VM in his virtualized network, BGP parameters (optional), the public address of the S2S VPN gateway on the on-premises site, and the on-premises address space reachable over the connection (if BGP is not provisioned).
The Windows Azure Portal screenshot snippets below illustrate these steps.
First, the customer specifies the subnet in his private address space where the S2S gateway should be located, and (optionally) enables BGP. The customer also specifies the address of the designated DNS server to be used by applications running in his virtualized network. This could be a server located in the customer’s virtualized network or in the on-premises network, or it could be a server made available by the service provider.
He next specifies the IP address space to be used for the virtualized network. This is typically from the customer’s private IP address space.
The customer then specifies the BGP parameters, which include the Autonomous System Number (ASN) of the virtualized network, the peer BGP IP address, and the ASN of the on-premises network.
This results in BGP configuration via VMM being triggered on the service provider side (see screenshot above). BGP may also be configured by the admin using a script as described here , which allows more detailed configuration.
Finally, the customer specifies a name for the on-premises site, the public IP address of the on-premises S2S gateway, and the IP addresses that are reachable on the on-premises site over the S2S connection. The last two parameters are required to set up S2S connection on-demand from the service provider side, and to determine which destinations in the extended network are in the on-premises side (if BGP is not enabled).
After completing these tasks, the customer deploys a S2S gateway at the on-premises site that connects to the service provider cloud. This could be an existing 3rd party edge device, or a Windows Server 2012 R2 S2S gateway.
In the former case, the customer uses vendor-specific commands to set up the S2S connection. In the latter case, customers can use the following script template to automate the configuration of the gateway:
|
############### Macros for RRAS Config on Ent GW ############################## $S2SDestination = "100.0.0.6" $IPv4Subnet = "172.23.90.4/32:100" $BGPPeerAddress = "172.23.90.10" $BGPAddress = "172.23.90.4" $BGP_PeerASN = 64522 $BGP_LocalASN = 64512 ################### Install S2S VPN on Ent GW ############################## New-VM -Name GW -MemoryStartupBytes 2.5GB -SwitchName Internet -VHDPath "E:GW-VMsGW.vhd" -Path "E:VM" Add-WindowsFeature -Name Remoteaccess -IncludeAllSubFeature -IncludeManagementTools ipmo remoteaccess Install-RemoteAccess -VpnType VpnS2S ############## Configure S2S VPN on Ent GW ############################## Add-VpnS2SInterface -Name "-Tunnel" -Protocol IKEv2 -Destination $S2SDestination -AuthenticationMethod PSKOnly -SharedSecret 111_aaa -Persistent -IPv4Subnet "172.23.90.10/32:100" -NumberOfTries 0 ############### Configure BGP on Ent GW ############################## #### Add BGP Router #### Add-BgpRouter -BgpIdentifier $BGPAddress -LocalASN $BGP_LocalASN #### Add BGP Peers #### Add-BgpPeer -Name "ServiceProviderSite1" -LocalIPAddress $BGPAddress -PeerIPAddress $BGPPeerAddress -PeerASN $BGP_PeerASN #### Add custom networks for advertisements to peers #### Add-BgpCustomRoute -Interface "Intranet" ## End Config Ent GW - ## |
Routing configuration on the customer premises
- As Figure 1 illustrated, a customer’s extended network consists of the on-premises network and the virtualized network in the service provider cloud.
- The customer’s private IP numbering plan applies to both of these networks – some of the customer’s IP subnets are located on-premises while others are in the cloud.
- To correctly route traffic between these subnets, the S2S gateway (from Microsoft or a 3rd party) at the customer premises must advertise routes to subnets situated within to the cloud gateway. Similarly, the cloud gateway must advertise routes to the subnets in the customer’s virtualized network.
- While BGP is used as the protocol to advertise and maintain these routes in the respective gateways, BGP itself should know what routes to advertise.
- On the cloud side, the routes to advertise are typically provisioned in the S2S gateway as part of creating virtual networks and S2S connection provisioning.
- There are two options for doing this on the customer premises (described below).
Manual configuration
- On-premises routes that should be advertised to the BGP peer can be configured manually in the on-premises gateway.
- Similarly, routes in the cloud that are reachable via the on-premises gateway must be manually configured in internal routers (see below).
- Manual configuration does have the disadvantage that routes have to be reconfigured whenever there are changes in subnets reachable on the on-premises or cloud side.
Route exchange using an internal routing protocol
- Figure 2 (see below) illustrates the topology of a customer site connecting to a hosted cloud through a 3 rd party edge, which also serves as a S2S gateway.
- The BGP protocol requires that these two sides have distinct Autonomous System Numbers (ASN), a parameter integral to the protocol.
Figure 2
- The customer site edge device learns the virtualized subnet routes (10.2.1.0/24) hosted in the cloud via BGP. It also advertises the on-premises subnet routes (10.1.1.0/24) to the service provider S2S gateway.
- The customer edge router learns on-premises internal routes through one of the following mechanisms:
- The edge device runs BGP with an internal router and learns internal routes (in this case, 10.1.1.0/24), or
- The edge device implements an Interior Gateway Protocol (IGP) such as OSPF, IS-IS, EIGRP, RIP, etc., and participates directly in internal routing.
In the former case, the internal router learns external routes (10.2.1.0/24) from the edge device, and it must distribute these routes to other on-premises routers using an IGP. In the latter case, the edge device itself must distribute the external routes to other on-premises routers using the IGP it runs.
A variation of this topology where Windows Server 2012 R2 gateway is deployed in the customer site is shown in Figure 3 (see below).
In this configuration, S2S VPN and BGP are terminated on Windows Server 2012 R2 S2S gateway deployed behind an edge firewall. In this case, since the gateway does not support distribution of routes between BGP and any IGP, only option 1 (noted above) is available for the S2S gateway to learn internal routes and distribute external routes.
Figure 3
Connecting multiple on-premises sites to the cloud
- A customer may choose to connect multiple on-premises sites to the service provider cloud. A single site may also have more than one S2S connection to the cloud.
- A requirement for VMM is that all such connections must terminate on the same S2S gateway in the cloud. With multiple S2S connections, BGP can be utilized to route traffic from the cloud to the on-premises along the best path.
- For instance, a particular S2S connection can be made preferable over others for certain destinations.
-
Also, if a connection goes down, traffic going over it can be automatically rerouted over other connections.
Scenario 2: Enabling direct Internet access for VMs in the virtualized network
Applications or services running in the customer’s virtualized network may access resources located on the Internet such as public DNS servers, web services, etc. This is a common requirement in many deployments. There are two options to enable this scenario:
- All traffic from the VM (including Internet-bound traffic) is tunneled to the customer’s premises via the S2S connection, and it is left up to the logic in the customer’s premises to correctly route Internet-bound traffic. From the customer’s point of view, this may not be the optimal way to route traffic to/from the Internet.
- The S2S VPN gateway identifies Internet-bound traffic and routes it directly to the Internet. The customer would most likely prefer this method.
The R2 release supports both options.
The Service provider can configure the NAT function in the S2S VPN gateway to enable direct Internet access from the customer’s virtualized network. The NAT function in the Windows Server 2012 R2 gateway is tenant-aware and allows VMs in multiple customer networks with overlapping IP addresses to access the same Internet IP address.
Customers can activate NAT for their virtualized networks using the Windows Azure Pack-based self-service portal. If S2S VPN and NAT are enabled for a virtual network, BGP enables learning of customer routes on the gateway so that only Internet traffic is NAT’ed.
Scenario 3: Enabling users to remotely access VMs in their virtualized network
There are often instances where resources available on a customer’s virtualized network need to be made accessible to users who are outside the customer’s premises.
Examples of this include disaster recovery, whereby the backup or recovery VM run in the cloud and need to be made accessible to users after a disaster situation. More routinely, a service running in the cloud may need to be made available to users who are working off-premises, including from public hotspots. Similarly, an administrator might want to login to a VM deployed in the cloud from anywhere.
To support this scenario, the service provider can enable access to VMs or services in a customer’s virtualized network from different devices over the Internet using the remote access VPN feature built into the Windows Server 2012 R2 S2S VPN gateway.
Users belonging to the customer’s organization can connect to their VMs or services in the cloud via the multitenant S2S gateway using industry standard IPsec/IKEv2 VPN connections from various devices like laptops, tablets and smartphones. If the devices are behind a firewall or router that does not allow IPsec traffic, SSTP (SSL) VPN can be used to connect to the gateway. SSTP VPN uses https (port 443) to connect to the gateway and can traverse firewalls that block any traffic that is not http/https.
The service provider can also enable remote access VPN for multiple customers on the same gateway with a single public IP address with all connections using port 443. Multiple customers can use overlapping IP addresses for VPN clients and the R2 multi-tenancy feature enables traffic isolation. All the VPN client connections are authenticated by the gateway directly or by using a RADIUS server in the service provider network.
After a user connects over VPN, he can access VMs in the appropriate virtualized network in the cloud as well as networks on-premises sites over S2S VPN (if provisioned). The service provider can easily enable multitenant VPN access in the S2S VPN gateway using a PowerShell script as described here . VMM support for this is not available in the R2 release.
The following figure shows the cmdlets used to enable VPN for customers Contoso and Fabrikam:
|
PS C:> Enable-RemoteAccessRoutingDomain -Name “Contoso” -Type Vpn PS C:> Enable-RemoteAccessRoutingDomain -Name “Fabrikam” -Type Vpn PS C:> Set-RemoteAccessRoutingDomain –Name “Fabrikam” –IPAddressRange 11.11.11.1, 11.11.11.200 –TenantName “Fabrikam” PS C:> Set-RemoteAccessRoutingDomain –Name “Contoso” –IPAddressRange 11.11.11.1, 11.11.11.200 –TenantName “Contoso” |
* * *
Windows Server 2012 R2 and System Center 2012 R2 provide a set of advanced capabilities for service providers to implement hybrid networking cost-effectively, reliably, and at scale. This includes multitenant S2S connectivity, NAT, and remote access VPN. In conjunction with the Windows Azure Pack, VMM, and PowerShell scripting – service providers can easily automate the on-boarding of customers, as well as set up and manage all hybrid networking functions.
Tomorrow we’ll cover another key part of a hybrid IT environment: Protecting your data with an enterprise-grade disaster recovery solution. It’s an under-served part of the industry, and I believe we have addressed it proactively with some very impressive features.
- Brad
NEXT STEPS:
To get even deeper on the topics covered in this post, check out the engineering posts and TechEd sessions included below.
-
Project MAT for Shift
From the popular Building Clouds blog, this post is an update on the very well received VM Migration toolkit. -
Networking for Cloud Services in Windows Server 2012 R2
In this video from TechEd Europe, session presenters cover the advances in core network infrastructure services (DNS, DHCP, and IPAM) in Windows Server 2012 R2, as well as how to implement these enhancements in private, public, and hybrid cloud environments. -
Deep Dive on Hyper-V network virtualization in Windows Server 2012 R2
In this video from TechEd North America, session presenters discuss how Hyper-V Network Virtualization is a key investment to enable workload mobility and SDN in the cloud. This session includes a deep dive into how Hyper-V Network Virtualization works. Also featured: How Windows Server 2012 R2 makes it easier than ever to enable customers to move their workloads, and for hosters to improve flexibility/automation/control across any cloud. -
How to design and configure networking in Microsoft VMM and Hyper-V
In this session from TechEd North America, the presenters discuss the comprehensive set of options and configuration settings for networking provided by Hyper-V VMM. -
What’s new in Windows Server 2012 R2 Networking
In this session from TechEd North America, you can learn more about how Microsoft has taken what it has learned from its global network of datacenters and applied it to the development of Windows Server 2012 R2. This session covers networking advancements, network infrastructure enhancements in IPAM, Secure Remote Access to better meet the needs of virtualized environments, and how we have advanced Software Defined Networking with in-box support for hybrid environments. -
What's New in IPAM in Windows Server 2012 R2
This technical overview covers role-based access control, virtual address space management, enhanced DHCP server management, external database support, upgrade/migration support, and enhanced Windows PowerShell support. -
Hyper-V Extensible Switch Enhancements in Windows Server 2012 R2
In this blog post, we detail the enhancements we have made to the Hyper-V extensible switch in Windows Server 2012 R2. -
What’s new in System Center 2012 R2, Virtual Machine Manager
This session from TechEd North America, presented by Vijay Tewari, discusses the new innovations in virtualization, storage, and networking. Vijay discusses the new capabilities of System Center 2012 R2 Virtual Machine Manager that enable new scenarios for customers – as well as enhancements to existing scenarios. Also discussed: How to use SDN to bring agility into cloud-based environments, and storage enhancements that enable customers to easily deploy enterprise-grade workloads. -
Test Lab Guide: Windows Server 2012 R2 Hyper-V Network Virtualization with System Cente...
Technical guidance with instructions for how to create the Windows Server 2012 R2 Network Virtualization with System Center 2012 R2 Virtual Machine Manager (VMM) – using computers running Windows Server 2012 R2. -
What's New in VMM in System Center 2012 R2
This detailed overview covers enhancements to Networking, VM management, Storage, Services, and Infrastructure. -
What’s New in Hyper-V Network Virtualization in R2
In this blog post, we go into details on the new capabilities of Hyper-V Network Virtualization in Windows Server 2012 R2. -
Network Virtualization Technical Details
Cloud-based datacenters can provide many benefits such as improved scalability and better resource utilization. To realize these potential benefits requires a technology that fundamentally addresses the issues of multi-tenant scalability in a dynamic environment. Hyper-V Network Virtualization was designed to address these issues and also improve the operational efficiency of the datacenter. -
Overview of Windows Azure Pack for Windows Server
This in-depth white paper evaluates how Windows Azure pack has been built on a foundation of Windows Server and System Center to deliver an enterprise class, cost-effective solution for multi-tenant cloud infrastructures services. Service Providers and enterprise customers can build customizable solutions using industry standard hardware, broad application platform support and open technologies. -
Evaluation Guide for System Center 2012 R2 and the Windows Azure Pack
Documentation reviewing how to set up and test an evaluation environment for tenant and administrator portals. -
Microsoft BGP Router configuration automation
Featured here is a PowerShell script that provides an easy-to-use automated interface for the configuration of BGP Router on Routing and Remote Access Server (both in Multi-Tenant and Single-Tenant modes) on a Microsoft Windows Server 2012 R2 system.