Azure App Service Web Apps has been around for many years now and has for a long time been a very reliable and performant product for running highly utilized workloads. One of the benefits of Azure App Service Web Apps is the scaling capability which lets you add additional compute resources like CPU and Memory (aka another VM to the web farm) on-demand. You can add additional compute power to your web site manually or using auto-scaling, as described here. How this works from an architecture is explained at the location linked in the following bullet point. The focus of this article however is concentrated on the File Server role which stores your code and static files.
When your website needs to add an additional Web Worker to handle additional load, the content that you originally published to the App Service which is stored on the File Server is used. This ensures that the files used on each Web Worker in your App Service Plan (ASP), aka web farm, are identical. Since your application code is referenced from the same location from each Web Worker the user experience will also be the same regardless of which Web Worker responds to the request. It is also true that when you update your application code, the update happens to a single location, I.e. the File Server. Because all the Web Workers reference that single location, they will begin delivering and responding with the newly updated code immediately, after loading the new code files. Updating code does cause some availability impact, the impact duration is dependent on what your application does at startup, therefore it is recommended to deploy to a staging slot and not directly into production.
An Azure App Service is organized into what is often referred to as a tenant or Scale unit. As described here, a Scale unit can have more than 1,000 servers which enable economy of scale through the reuse of hardware and infrastructure. If you want your Scale unit you can provision what is called an App Service Environment (ASE). A summarized example which illustrates the Front End role and Web Worker that are situated inside of a Scale unit are shown in Figure 1.
Figure 1, front end and web worker inside an Azure App Service scale unit
From Figure 1 you should recognize that the File Server is a shared role in the Scale unit. There are for certain more than a single instance of the File Server role, however it is not dedicated to the Web Worker. This kind of storage link is called persistent share that is most commonly and historically implemented using a symlink. In that scenario there is a protocol named SMB which was exploited some years ago by viruses named EternalBlue and WannaCry, those vulnerabilities have long sense been eradicated, but it is interesting how SMB and this symlink persistent data share are connected. The reason is that file changes are not monitored through a symlink share, therefore, in order to be notified of file changes there needs to be some service monitoring them. That service is the Service Message Block (SMB) protocol. Without SMB you could conceivably deploy new code and nothing happens, I.e. there is no File Change Notification (FCN) that instructs the process to restart with the new version of the code. These days it is expected that the platform loads new files and restarts the process automatically and most are oblivious to the existence of SMB and FNC, however, they are fundamental OS features. You can capture SMB and FCN trace logs using NETSH or NETMON.
- The SMB vulnerability existed only in SMBv1 which is no longer installed on Windows servers
- When you mount a drive it is treated just like any other system drive like the very common C:\ drive
Now, back to the main topic; there is a scenario known as the ‘noisy neighbor’ which happens when one (or more) applications running on the Web Workers in a Scale unit consume too much of the capacity on the File Server causing impact on other Web Workers in the same Scale unit. Web sites which perform a lot of I/O operations are notorious for being noisy neighbors. It does not happen often, it typically does not last very long, but it can cause latency and transient issues, it is also something which is monitored and triggers action from the platform team. To remove or reduce the impact of a noisy neighbor or if your application itself performs a large amount of I/O operations that could cause latency, you might consider mounting Azure Storage as a local share in App Service as described here and perform those operation in that location instead of on the default, shared File Server role, Figure 2 illustrates how that might look.
Figure 2, front end and web worker inside an Azure App Service scale unit with mounted Azure file shares
Notice that all of the applications on the Web Workers continue to have access to the File Server role even after adding a mount. The directories which store your log files, deployment logs, event viewer logs, etc, still get placed into the /home directory hierarchy. The example shown in Figure 2 is also known as Bring Your Own Storage (BYOS) which gives you the ability to read application files and read/write data to an Azure File share instead of the File Server persistent share.
What is and why use BYOS?
BYOS is a high available and scalable solution for customers to remove dependency on the File Server role. The solution is available for both Windows and Linux apps. One aspect of a Platform-as-a-Service (PaaS) offering is that you are not responsible for the servicing and maintenance of the server on which your code runs, instead this is the responsibility of the cloud service provider. When maintenance happens on the File Server role (which is a VM) that action will more than likely result in the restarting of your application. However, if your application code is located on the mounted BYOS share, that restart can be avoided. Benefits gained from the WEBSITE_DYNAMIC_CACHE application setting are also realized when using BYOS. The WEBSITE_DYNAMIC_CACHE application setting is enabled by default and stores recently accessed files on the local machine into the %SYSTEMDRIVE%\local\DynamicCache directory. That application setting and the use of a Premium tier Azure Storage account should be sufficient for a majority of Azure App Service customers. Finally, customers who are using RunFromPackage, which is a supported scenario, can also benefit from BYOS.
A few points you need to consider when choosing a BYOS storage solution:
- The backup and restore Azure App Service experience is not available. You need to manage that scenario yourself using the features provided from Azure Storage. Here is some information about that to get you started.
- Connectivity to the Azure Storage Account is managed by a SAS key. This adds a bit of management overhead as you need to manage the integrity and security of the key
- There is currently no solution for you to migrate your existing Azure App Service Web App to BYOS. If your Web App is currently pulling content from the FIle Server the only option you have to change to a BYOS is to rebuild and configure your application.
- Both Azure Blob storage and Azure Files are supported. However, a BYOS configured with Azure Blob storage is read-only, which Azure Files is read/write.
- FTP/FTPS is not supported through BYOS through Azure App Services
- Make sure the Azure Storage account is in the same region as the Azure App Service to avoid latency
Optimal Azure App Services Storage guidelines
Here are some recommendations for storing content
- Do not enable Failed Request Tracing or Detailed error logging on the production slot for long durations. At one time, this feature was disabled by the platform after 12 hours, however, this is no longer the case. If not disabled it will capture these very verbose logs, which is resource intensive and carries with it heavy I/O operations. If required for troubleshooting an issue, enabled it, but do not forget to disable it once testing is complete.
- Avoid writing logs into the wwwroot directory
Figure 3, the wwwroot directory of an Azure App Service Web App as seen in FileZilla
- Avoid making changes to any files that exist in the wwwroot directory
- Actively manage and monitor temp file usage located at either D:\local\temp or C:\local\temp
Figure 4, site folder and temp folder locations as seen through KUDU
- If you mount an Azure File share the associated Azure Storage Account should be on Premium tier which has a 5ms response time versus 30ms on Standard tier when the file is not cached.
These actions and activities will help you prevent performance and availability issues caused by storage capacity.
Other helpful links
Here are some links which might be helpful for continued learnings
- Bring Your Own Storage (BYOS) Azure App Service map directory path
- Mount Azure Storage as a local share in App Service
- Bring Your Own Storage in Azure Web App
- Configure Bring Your Own Storage (BYOS) for Application Insights Profiler and Snapshot Debugger
- FAILED TO INITIALIZE RUN FROM PACKAGE.txt
- Azure App Service and Reliability
- Analyze NETSH traces with Wireshark or Network Monitor, convert ETL to CAB
- Advanced Troubleshooting Server Message Block (SMB)