Way to minimize the impact of Allocation Failure issue in Cloud Service Extended Support

Allocation Failure is a common issue for Cloud Service Extended Support. The cause of this issue is explained in our official document. The best solutions are as documented: Redeploy to a new Cloud Service or Delete Swappable Cloud Services. But for both, the downtime is unavoidable. For real user scenario, it’s almost impossible and very harmful to completely delete a Production environment deployment and recreate it as it will cause a huge impact.

This blog will mainly talk about the way to mitigate the Allocation Failure issue by switch the request to newly created Cloud Service in order to minimize the impact.

P.S. The blog will use a CSES with public IP address as example to explain. If internal Load Balancer is used which means CSES doesn’t have any public IP address, this blog can also work after adjusting the IP address with the private frontend IP address.

Overall Idea:

The nature of this blog is to create a new Cloud Service Extended Support with the new, expected settings (such as new VM size, new instance number etc.) then reroute the traffic to them. For this purpose, a domain name, no matter custom domain name or FQDN (Fully Qualified Domain Name) configured in Public IP address resource, is necessary.

Next two parts will explain the solution based on the domain name types.

If custom domain name is used:

When users visit the custom domain (For ex: abc.com), it will either use a CName record which points to the FQDN of the CSES or use an A record which points to the IP address of the CSES.

In this scenario, the solution will be straightforward:

• Create a new CSES (not swappable with original CSES) with new public IP address, new/same Virtual Network, same project and same configuration such as extensions if there are.
• Test all functionality and make sure that the new CSES is good enough to work as the real production environment.
• Modify the CName or A record setting to map the custom domain name to the new CSES FQDN or IP address. (The way of modifying this setting depends on which domain provider you’re using. It should be easy to find guidelines from the domain provider)
• Make sure that the client side works well with new CSES and no more traffic is routed to original CSES (avoid impact of DNS cache or delay on client side), then remove the original CSES and its related resources (The last part of this blog will explain how to verify if there is traffic routed into CSES)

If FQDN is used:

If the FQDN configured in Public IP address is used and custom domain is not used which means users are using domain name like CSES.region.cloudapp.azure.com to reach the CSES, the idea will be similar to the first scenario with A record.

Attention! This may cause a short downtime because the manual switch of FQDN configured on public IP addresses resource needs time to be finished. The downtime usually will be around 1 minute.

What users need to do here are:

• Create a new CSES (not swappable with original CSES) with new public IP address with DNS name, new/same Virtual Network, same project and same configuration such as extensions if there is.
• Test all functionality and make sure that the new CSES is good enough to work as the real production environment.
• Note down the DNS name of original public IP address (such as CSES.region.cloudapp.azure.com), then change it to a whatever different value (such as originalCSES.region.cloudapp.azure.com)
• Change the DNS name of new public IP address to the DNS name of original public IP address (CSES.region.cloudapp.azure.com)
• Make sure that the client side works well with new CSES and no more traffic is routed to original CSES (avoid impact of DNS cache or delay on client side), then remove the original CSES and its related resources (The last part of this blog will explain how to verify if there is traffic routed into CSES)

How to verify incoming traffic of CSES

This can be verified from the metrics page of CSES resource. Please select Network In Total as value of Metric.

Attention! Even if the CSES is not used, there might still be a little incoming network traffic due to the design of CSES. There should be a clear difference between the network traffic usage by production environment and the design.

Summary:

The Allocation Failure issue itself is unavoidable as it’s a kind of design limitation. It can happen to every user. But the above solutions should be able to minimize the impact of this issue.

Per test result, the solution for scenario with custom domain should be with zero downtime and the downtime of FQDN scenario should be less than 1 minute.