New Azure Feature Pack Release Updating HDInsight Support

First published on MSDN on Mar 02, 2017
Dear Customers,

I'm happy to announce that we have released a new version of SSIS Azure Feature Pack (AFP) highlighting on https://azure.microsoft.com/en-us/services/hdinsight/ support upgrade. The download links are as follows:

• https://www.microsoft.com/en-us/download/details.aspx?id=47367


• https://www.microsoft.com/en-us/download/details.aspx?id=47366


• https://www.microsoft.com/en-us/download/details.aspx?id=49492


• https://www.microsoft.com/en-us/download/details.aspx?id=54798

Since the introduction of HDInsight support in AFP, there have been profound changes made to the HDInsight service on Azure. This new release aims to support these changes.

1. Change from classic deployment model to Azure Resource Manager (ARM) model. This is an on-going effort for all Azure resources, not just limited to HDInsight. It changes the way Azure resources are managed. More details can be found https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-deployment-model . Concerning our case, this affects the creation/deletion of HDInsight clusters which are done by the https://msdn.microsoft.com/en-us/library/mt146774.aspx / https://msdn.microsoft.com/en-us/library/mt146778.aspx https://msdn.microsoft.com/en-us/library/mt146778.aspx , respectively. In previous versions, the classic certificate-based https://msdn.microsoft.com/en-us/library/mt146773.aspx is used by the tasks for resource-management authentication. In this new release, a new https://msdn.microsoft.com/en-us/library/mt807533.aspx is introduced for this purpose, and the two control flow tasks are updated to use this new connection manager in place of the original one. Currently, only https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal authentication is supported by the new connection manager.


2. Change from Windows-based clusters to Linux-based. In the beginning, only Windows-based clusters are supported. Nowadays, however, Windows-based clusters are being deprecated, and Linux-based ones are taking the place. As explained by the HDInsight team: " For continued investment on the open source big data technologies, future releases of HDInsight will be available only on Linux OS. There will not be any future release of HDInsight on Windows OS. The last release of HDInsight on Windows was HDI 3.3. The support for HDI 3.3 expired on 06/27/2016 and it will be deprecated on 07/31/2017. " Following this trend, since this release, the Azure HDInsight Create Cluster Task creates Linux-based clusters instead of Windows-based ones as done in previous versions. Compared to Windows-based clusters, Linux-based require two extra properties in Azure HDInsight Create Cluster Task , namely SshUserName and SshPassword which are used to remote-connect to the clusters via SSH.


3. Introduce the new https://msdn.microsoft.com/en-us/library/mt807532.aspx . This new connection manager is used by https://msdn.microsoft.com/en-us/library/mt146771.aspx / https://msdn.microsoft.com/en-us/library/mt146781.aspx https://msdn.microsoft.com/en-us/library/mt146781.aspx to identify the target HDInsight cluster to run the script and supply authentication information.


4. Extra properties in Azure HDInsight Hive/Pig Task to fetch script execution outcome and error logs. The extra properties include an https://msdn.microsoft.com/en-us/library/mt146780.aspx and a blob container name which are used to specify the default storage account and container associated with the cluster, respectively.

Regards,
Lingxi