There are multiple scenarios where file data needs to be transferred from Mainframe and Midrange Systems to Azure. Few of them could be
Couple of major hurdles in these architectures are movement of data from one system to another and second major hurdle would be codepage and datatype conversion. As Mainframe/Midrange works on a different codepage and has many custom datatypes like computational fields, which are not straight forward for conversion, Microsoft first party tool, Data Provider for Host Files from Host Integration Server (HIS) provides an easy way to unpack data types and convert from Mainframe EBCDIC Codepage to ASCII.
Along with HIS, we now have a new custom Azure Data Factory (ADF) connector which can help customers to convert Mainframe data on Azure. As ADF is a low code, no code solution, there would not be much development needed from the customer to achieve the conversion on Azure. This blog talks about how we can setup and execute the new custom ADF connector.
The new solution to this would take a file in Blob as input along with the copybook file (Metadata file). With the help of Metadata file, ADF custom connector can convert the data from Mainframe format to ASCII format.
High Level architecture of the solution would look as below:
Figure-1: High Level architecture
Figure-2: Detailed Architecture
Pre-requisite:
• Transfer Mainframe data in binary format from Mainframe to Azure Blob Storage container.
• Corresponding Copybook (Metadata file) in the same folder as the data file.
• Windows Virtual Machine with SHIR installed on it
• HIS product installed on the SHIR machine.
• ADF configured to corresponding SHIR machine.
Setup involves two stages.
• Stage1: Setup the custom connector on the SHIR machine.
• Stage2: Setup a pipeline which uses the newly setup custom Connector.
• Setup SHIR on the VM using one of the many methods highlighted in this document.
• Search for “Microsoft Integration Runtime” and stop the server.
• Please write to datasqlninja@microsoft.com to get the package.
• Create a new folder named “CustomPlugins” under the folder ‘C:\Program Files\Microsoft Integration Runtime\5.0’
• Create a new folder named “Test” under the folder ‘C:\Program Files\Microsoft Integration Runtime\5.0\CustomPlugins’
• Place a json file named “CustomPlugins.json” provided in the package into the test folder.
• Create a folder named “EBCDICtoASCII/1.0.0.0” under the test folder.
• Copy contents of package provided into this folder. Effectively all the contents should be copied into folder “C:\Program Files\Microsoft Integration Runtime\5.0\CustomPlugins\Test\EBCDICtoASCII\1.0.0.0”
• Start the “Microsoft Integration Runtime” software which was stopped initially.
Create a new Pipeline in the ADF and give it a new name.
Now drag and drop a new Copy Activity in the pipeline
Search for "Mainframe File Converter", Click on the connector
Give the linked service an appropriate name, choose the integration runtime, then choose the right connection string that has been put into Azure key vault. In the key value you will need to store your blob SAS string so that ADF will have access to the Blob Container. Read more about how to create a SAS Key here. Here "CustomConnector" is the name of integration runtime. You might have given a different name for the same. Please choose the integration runtime which points to SHIR which was configured earlier.
Enter the input container, folder and file name in the given space This is where the binary version of Input data file is present.
Enter the copybook name of the data file which was provided earlier. Also make Max concurrent connection as 1.
Move to the Sink tab and click on new
Search for Blob and click on Blob Storage and click continue
Select type Binary, then give details of Azure Integration runtime, along with details of where the output needs to be stored
The pipeline is now ready to be execute. Just click on Debug to start running the pipeline. Once it is completed, it will give all the necessary stats in ADF UI
You should now see the desired output in sink configuration with comma separated fields.
If you have feedback or suggestions for improving this asset, please contact the Azure Databases SQL Customer Success Engineering Team. Thanks for your support! Note: For additional information about migrating various source databases to Azure, see the Azure Database Migration Guide.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.