First published on MSDN on Mar 02, 2017
Dear Customers,
I'm happy to announce that we have released a new version of SSIS Azure Feature Pack (AFP) highlighting on
HDInsight
support upgrade. The download links are as follows:
Since the introduction of HDInsight support in AFP, there have been profound changes made to the HDInsight service on Azure. This new release aims to support these changes.
-
Change from classic deployment model to Azure Resource Manager (ARM) model.
This is an on-going effort for all Azure resources, not just limited to HDInsight. It changes the way Azure resources are managed. More details can be found
here
. Concerning our case, this affects the creation/deletion of HDInsight clusters which are done by the
Azure HDInsight Create Cluster Task
/
Azure HDInsight Delete Cluster Task
, respectively. In previous versions, the classic certificate-based
Azure Subscription Connection Manager
is used by the tasks for resource-management authentication. In this new release, a new
Azure Resource Manager Connection Manager
is introduced for this purpose, and the two control flow tasks are updated to use this new connection manager in place of the original one. Currently, only
service principal
authentication is supported by the new connection manager.
-
Change from Windows-based clusters to Linux-based.
In the beginning, only Windows-based clusters are supported. Nowadays, however, Windows-based clusters are being deprecated, and Linux-based ones are taking the place. As explained by the HDInsight team: "
For continued investment on the open source big data technologies, future releases of HDInsight will be available only on Linux OS. There will not be any future release of HDInsight on Windows OS. The last release of HDInsight on Windows was HDI 3.3. The support for HDI 3.3 expired on 06/27/2016 and it will be deprecated on 07/31/2017.
" Following this trend, since this release, the
Azure HDInsight Create Cluster Task
creates Linux-based clusters instead of Windows-based ones as done in previous versions. Compared to Windows-based clusters, Linux-based require two extra properties in
Azure HDInsight Create Cluster Task
, namely
SshUserName
and
SshPassword
which are used to remote-connect to the clusters via SSH.
-
Introduce the new
Azure HDInsight Connection Manager
.
This new connection manager is used by
Azure HDInsight Hive Task
/
Azure HDInsight Pig Task
to identify the target HDInsight cluster to run the script and supply authentication information.
-
Extra properties in Azure HDInsight Hive/Pig Task to fetch script execution outcome and error logs.
The extra properties include an
Azure Storage Connection Manager
and a blob container name which are used to specify the default storage account and container associated with the cluster, respectively.
Regards,
Lingxi