Troubleshooting Storage Firewall Issues
Published Dec 07 2020 02:48 AM 16.7K Views
Microsoft

In this blog we will look at some common issues that we face using storage accounts with Firewalls and Virtual Networks enabled. We have enabled storage diagnostics logs on the storage account, and we will use the same to troubleshoot some of the issues.

 

You have enabled Firewalls and Virtual networks on your storage account and allowed access to the storage account only from specific Virtual Network(s) (VNet).

image.png

 

Scenario 1:

You are not able to access your storage account using Portal from an on-premises network (not part of the Azure VNet) or over the internet.

 

image.png

 

Actions:

  1. The error message we are getting is Authorization Error when accessing the storage account from our on-premises system.
  2. We will download the storage diagnostics logs and look for additional information on this error.
    image.png
  3. In the log file we will look up the ‘Storage Request ID’ that we see in the error, which is ‘c18737e5-b01e-000a-04c5-b9b483000000’ in this case.
    image.png
  4. From the logs we can see that the request failed due to ‘SASIpAuthorizationError’ and we can see the originating IP address as well.
    image.png
  5. As we have allowed access to our storage account only from specific VNet, we need further authorize our client IP Address as well.
  6. For that we will navigate back to ‘Firewalls and virtual networks’ and under Firewall, we will add our client IP address and click Save.
    image.png
  7. Once done we are now able to access the storage account containers contents.
    image.png

 

Scenario 2:

You are not able to access your storage account from a Virtual Machine, which is part of the VNet, already authorized in storage accounts Firewall and virtual networks.

 

When trying to download a file, we see the following error message.

image.png

 

Actions:

  1. For this issue we will use the storage diagnostics logs enabled on our storage account. We will navigate to the $logs container in our storage account and download log files.
    image.png
  2. We have converted the .log file to a CSV using this PowerShell script for easier analysis. https://gist.github.com/ajith-k/aa69feb862a4816d0b4df09fae8aad11
  3. As we were trying to download a blob, we will filter the logs and look for ‘GetBlob’ operations and look for failed requests.
    image.png
  4. Below are the details of the error message that I have extracted from logs.

    Transaction Start Time

    2020-11-13T14:46:26.5411013Z

    REST Operation Type

    GetBlob

    Request Status

    OAuthIpAuthorizationError

    HTTP Status Code

    403

    Authentication type

    bearer

    Service Type

    blob

    Request URL

    https://storageaccount.blob.core.windows.net:443/testcontainer/Image1.png

    Request ID

    6c736153-f01e-0024-16cb-b9e694000000

    Client IP

    10.1.3.4:50265

    User Agent

    Azure-Storage/2.0.0-2.0.1 (Python CPython 3.6.8; Windows 10) AZURECLI/2.11.1 (MSI)

    User Object ID

    9e1xxxxx-xxxx-xxxx-xxxx-xxxxxx786d11

  5. To confirm that this error indeed originated from our VM, we can verify the IP Address. For that we can simply run ipconfig command in our Virtual Machine, or on Azure Portal, we can go to the VNet this VM belongs to and check under Connected devices.
  6. The Request Status field denotes that this request was failed due to IP Authorization error. OAuth prefix denotes the authentication method used for this request. HTTP Status code denotes 403 which means unauthorized access.
  7. Next, we need to verify that the subnet in which this VM is assigned to is also allowed in the storage firewall.
  8. The VM belongs to testVNet1 and subnet is subnet3.
    image.png
  9. Under storage accounts, Firewalls and virtual networks we can see that only subnet0 is allowed to access the storage account.
    image.png
  10. We need to authorize subnet3 and enable Storage Endpoint on that subnet. If storage endpoint is not enabled, Portal will show a message and give the option to enable the storage endpoint.
  11. Once enabled, we can add subnet3 to the storage accounts firewall.</snap<
    image.png
  12. Once the new firewall rules are propagated, we can go back to our VM and try to download the blob again and it runs successfully.
    image.png

 

Scenario 3:

You are trying to add a VNet and its subnets to storage accounts firewall. However, you are getting NetworkSourceDeleted error.

image.png

 

Actions:

The error message in this case is very self-explanatory. The subnet ‘subnet1’ under testvnet1 is required to be removed from storage accounts named in the error message.

 

Let us understand why this error occurs. We have a Virtual Network setup as below and all these subnets are added to a storage accounts firewall:

VNet1   Storage1 Firewall
⇒subnet0   VNet1\subnet0
⇒subnet1   VNet1\subnet1
⇒subnet2   VNet1\subnet2

 

Now, if you delete a subnet from the virtual network, that subnet gets marked as NetworkSourceDeleted in the storage account.

 

VNet1   Storage1 Firewall
⇒subnet0   VNet1\subnet0
⇒subnet1 -deleted   VNet1\subnet1 -NetworkSourceDeleted
⇒subnet2   VNet1\subnet2

 

We create another subnet having the same name as the one which was deleted earlier. The previously deleted subnet1 is still marked as ‘NetworkSourceDeleted’ under Storage1 firewall.

 

VNet1   Storage1 Firewall
⇒subnet0   VNet1\subnet0
⇒subnet1 -new   VNet1\subnet1 -NetworkSourceDeleted
⇒subnet2   VNet1\subnet2

 

If we try to add the new ‘subnet1’ to any other storage accounts firewall, we get ‘NetworkSourceDeleted’ error. To resolve this:

 

  1. We go to Firewalls and virtual networks under storage accounts mentioned in the error and remove subnet1 from the Virtual networks allowed.
  2. Then if we try to add the subnet1 in any other storage account, it will not throw the error and complete the operation successfully.
    image.png
    image.png
1 Comment
Version history
Last update:
‎Dec 07 2020 02:54 AM
Updated by: