Scenario:
Let’s say, your application makes use of Azure Storage Account for uploading/downloading blobs. However, you are seeing that there is latency in either uploading the blob or downloading it. Though, there has been no change in your application, you are unable to figure out what is causing the issue. You are unsure of whether the latency is part of the application or from Azure Storage Account.
Now, let’s see how we can isolate the latency issue and understand whether the latency is from Storage account or Client Application.
Before, we understand how to troubleshoot the latency issue, let us understand what Server Latency and E2E Latency is:
The first starting point to identify any latency issue, is to check the metrices available on the Portal for the storage account. Navigate to the storage account -> Go to the Metrices tab -> Select the metric values depending on which Storage Service you are using. For example, I am making use of the Blob Service for selecting the metric values:
In usual scenarios, there should be less difference between the Sever Latency and E2E Latency. If you see that the E2E latency is significantly higher than the Server Latency, then it needs investigation as to why there is client-side latency.
If you have the Storage Analytical Logging enabled on the storage account, we can make use of these logs to find whether there is server latency or E2E latency. You can know more about the Storage Analytical Logging by referring to this link. The storage analytical logs are saved in the $logs container and can be accessed using the Storage Explorer tool. Please refer to the below screenshot for reference:
Now, in the storage analytical logs, we can see the E2E latency and Sever latency value for each operation done on the storage account. A log sample is as below:
In the logs, we can see that there is high E2E latency as compared to the Server Latency.
Thus, by making use of the metrices available on Portal or Storage Analytical Logs, we can understand whether the latency seen is from the client-side or from the Storage Server side.
Let us assume that there is high E2E latency and low Server Latency. Now, how are we going to isolate this issue further i.e. how to understand if the latency is because of the client application/VM’s performance or/and there is issue with the network.
The common attributes that can be checked to see if there is performance issue with the client application/VM are:
If you observe any of the above parameter while running your application, you will have to consider increasing your VM’s size. Try to scale your VM and check if there is any latency issue while uploading/downing blob to storage account.
If you find any performance issues with the application, you can refer to the performance checklist mentioned in the link. Note that the checklist is for the .NET applications.
You can also face the latency issue caused due to the Azure DC region from your application's location. For isolating the Azure DC latency issue, you can follow the below steps:
You can also investigate the issue from network side. Having a good VM configuration is not enough if you are having low bandwidth network. The operations performed on storage account with low bandwidth network will have latency issue. Thus, we need to investigate if there is low network bandwidth which is causing high E2E latency.
For this, you can make use of various tools like PSPing, TCPing or simply take a traceroute from your client machine to the storage account. Additionally, you can also make use of the AzCopy command to see the throughput you are getting while uploading/downloading the blob from Storage Account.
You can refer to the below steps:
3. Tracert command: You can get the trace route from your client machine to the storage account. You can execute the below command to get the details:
tcping.exe <yourstorageaccountname>.blob.core.windows.net.
If you observe any issue with network, then you will have to consider moving your application closer to region where the Storage Account is hosted. Let say that your application is hosted on Azure VM in Central US and the storage account it is accessing is in East US. To have better network performance, you will have to consider placing your application in the same region as that of storage account to avoid the network latency.
Hope this article helps you in isolating the latency issue and helps you in taking appropriate steps to mitigate it.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.