Trigger ADF pipeline using Storage event trigger over private network.

[Sachin215]

Project Technology: Azure function, ADF, Azure Synapse, ADLS

Issue description: Customer has a strict regulatory compliance requirement, due to which they need to block all outbound (public endpoint) connections. Hence, most of our products were not able to provide the expected result as we have a dependency on public endpoints.

Summary:

1. The customer was not able to download PowerShell modules from PowerShell gallery in the Azure function due to outbound restrictions. However, we gave a suggestion to manually download and upload the files to Azure function via VS-Code. However, it did not work either.
2. As a result, we used PowerShell command line from user desktop instead of using VS-Code.
3. Now when we tried to access the ADF from Azure function, Boom! It failed.

4. To validate the access token, we executed MSI validator and came with the below error.

5. Reason for all these errors was, AZ PowerShell was trying to connect to management.azure.com in order to get the Oauth2 token. However, as per the bank’s regulations, access to any public endpoints was not allowed which basically stalled the project.
6. To get around this and access the storage behind the firewall/private endpoints/Private links, we proposed a solution to use Managed identity and REST API which enabled the function to grab bearer token without public endpoint access and REAST API was able to use that bearer token to access Azure storage.

Code if customer is using system assigned managed identity.

$resourceURI = "https://functeststorageacc01.queue.core.windows.net/"

$tokenAuthURI = $env:IDENTITY_ENDPOINT + "?resource=$resourceURI&api-version=2019-08-01"

$tokenResponse = Invoke-RestMethod -Method Get -Headers @{"X-IDENTITY-HEADER"="$env:IDENTITY_HEADER"} -Uri $tokenAuthURI

$accessToken = $tokenResponse.access_token

Code if customer is using user assigned managed identity.

$resourceURI = "https://functeststorageacc01.queue.core.windows.net/"

$tokenAuthURI = $env:IDENTITY_ENDPOINT + "?resource=$resourceURI&api-version=2019-08-01&client_id=$env:AZURE_CLIENT_ID"

$tokenResponse = Invoke-RestMethod -Method Get -Headers @{"X-IDENTITY-HEADER"="$env:IDENTITY_HEADER"} -Uri $tokenAuthURI

$accessToken = $tokenResponse.access_token

Code to use bearer token to access Azure storage.

$version = "2017-11-09"
$header = @{
    Authorization = "Bearer $accessToken"
    'x-ms-version' = $version
}

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

$QueueMessage = "This is test message#1 "
$body = "<QueueMessage><MessageText>$QueueMessage</MessageText></QueueMessage>"
$item = Invoke-RestMethod -Method POST -Uri https://storazaarfdevbtgt00003.queue.core.windows.net/test2/messages -Headers $header -Body $body -ContentType "application/json"

7. Even though we had the bearer token and we were using REST API but still we were not able to trigger the ADF pipeline because in order to perform any operations using REST API to ADF would require access to Azure management plane (management.azure.com) which is not allowed in the bank’s environment.
8. Here we proposed a solution to trigger the ADF pipeline through storage event trigger using managed private endpoints, so ADF was able to read the storage over the private endpoints without needing to go out to public endpoints.

9.After creating all the above steps we were able to trigger the ADF using Storage Event.

Co-Author: Umesh Panwar (Apps & Infra CSA)