Transform data in ADF with Azure Cognitive Services

Published May 06 2022 08:15 PM 3,265 Views
Microsoft

We’ve added two new templates to the Template Gallery to help you easily perform complex transformations on your data with Azure Cognitive Services.

 

  1. PII detection and masking

With Azure Cognitive Services, you can now detect and mask PII in your data. Learn more here.

Noelle_Li_2-1651250789349.png

 

 

This template calls the Cognitive Services API using the external call transformation in a dataflow. The data flow includes transformations to create the appropriate request body before masking PII data row-by-row with the external call transformation.

 

Instructions on how to use the template can be found here.

 

  1. Extract data from PDF source

With Azure Form Recognizer, you can easily extract tables and text from many document sources using prebuilt or custom models.  Learn more here.

Noelle_Li_1-1651250768094.png

 

This template calls the Azure Form Recognizer API to extract data from a PDF source using a web activity. Then, using mapping data flow transformations, the extracted data is consolidated into a readable form that lands in a sink of your choice. You also have the option to reuse the dataflow logic by selecting the transformations and creating a new flowlet.

Instructions on how to use the template can be found here.

 

 

We are always open for feedback! You can provide suggestions or vote via our Ideas forum Data factory · Community (azure.com).

%3CLINGO-SUB%20id%3D%22lingo-sub-3298972%22%20slang%3D%22en-US%22%3ETransform%20data%20in%20ADF%20with%20Azure%20Cognitive%20Services%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3298972%22%20slang%3D%22en-US%22%3E%3CP%3EWe%E2%80%99ve%20added%20two%20new%20templates%20to%20the%20Template%20Gallery%20to%20help%20you%20easily%20perform%20complex%20transformations%20on%20your%20data%20with%20Azure%20Cognitive%20Services.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EPII%20detection%20and%20masking%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%20style%3D%22%20padding-left%20%3A%2030px%3B%20%22%3EWith%20Azure%20Cognitive%20Services%2C%20you%20can%20now%20detect%20and%20mask%20PII%20in%20your%20data.%20%3CA%20title%3D%22Cognitive%20Services%20PII%20Detection%20Docs%22%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsearch%2Fcognitive-search-skill-pii-detection%23%3A~%3Atext%3DThe%2520PII%2520Detection%2520skill%2520extracts%2520personal%2520information%2520from%2Cthat%2520exceed%252020%2520documents%2520per%2520indexer%2520per%2520day.%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ELearn%20more%20here%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22Noelle_Li_2-1651250789349.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F368045iD5FFA2FA2B94713A%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22Noelle_Li_2-1651250789349.png%22%20alt%3D%22Noelle_Li_2-1651250789349.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22%20padding-left%20%3A%2030px%3B%20%22%3EThis%20template%20calls%20the%20Cognitive%20Services%20API%20using%20the%20external%20call%20transformation%20in%20a%20dataflow.%20The%20data%20flow%20includes%20transformations%20to%20create%20the%20appropriate%20request%20body%20before%20masking%20PII%20data%20row-by-row%20with%20the%20external%20call%20transformation.%3C%2FP%3E%0A%3CP%20style%3D%22%20padding-left%20%3A%2030px%3B%20%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22%20padding-left%20%3A%2030px%3B%20%22%3EInstructions%20on%20how%20to%20use%20the%20template%20can%20be%20found%20%3CSTRONG%3E%3CA%20title%3D%22PII%20detection%20and%20masking%20Template%20Guide%22%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fsolution-template-pii-detection-and-masking%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehere%3C%2FA%3E.%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3COL%20start%3D%222%22%3E%0A%3CLI%3EExtract%20data%20from%20PDF%20source%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%20style%3D%22%20padding-left%20%3A%2030px%3B%20%22%3EWith%20Azure%20Form%20Recognizer%2C%20you%20can%20easily%20extract%20tables%20and%20text%20from%20many%20document%20sources%20using%20prebuilt%20or%20custom%20models.%20%26nbsp%3B%3CA%20title%3D%22Form%20Recognizer%20Docs%22%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fapplied-ai-services%2Fform-recognizer%2F%3Fbranch%3Drelease-build-cogserv-forms-recognizer%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ELearn%20more%20here.%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22Noelle_Li_1-1651250768094.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F368043iFE5B7BDA22897896%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22Noelle_Li_1-1651250768094.png%22%20alt%3D%22Noelle_Li_1-1651250768094.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22%20padding-left%20%3A%2030px%3B%20%22%3EThis%20template%20calls%20the%20Azure%20Form%20Recognizer%20API%20to%20extract%20data%20from%20a%20PDF%20source%20using%20a%20web%20activity.%20Then%2C%20using%20mapping%20data%20flow%20transformations%2C%20the%20extracted%20data%20is%20consolidated%20into%20a%20readable%20form%20that%20lands%20in%20a%20sink%20of%20your%20choice.%20You%20also%20have%20the%20option%20to%20reuse%20the%20dataflow%20logic%20by%20selecting%20the%20transformations%20and%20creating%20a%20new%20flowlet.%3C%2FP%3E%0A%3CP%20style%3D%22%20padding-left%20%3A%2030px%3B%20%22%3EInstructions%20on%20how%20to%20use%20the%20template%20can%20be%20found%20%3CSTRONG%3E%3CA%20title%3D%22Extract%20data%20from%20PDF%20Template%20Guide%22%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fsolution-template-extract-data-from-pdf%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3Ehere%3C%2FA%3E.%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%20style%3D%22%20padding-left%20%3A%2030px%3B%20%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22%20padding-left%20%3A%2030px%3B%20%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22%20padding-left%20%3A%2030px%3B%20%22%3E%3CSTRONG%3E%3CSPAN%3EWe%20are%20always%20open%20for%20feedback!%20You%20can%20provide%20suggestions%20or%20vote%20via%20our%20Ideas%20forum%26nbsp%3B%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Ffeedback.azure.com%2Fd365community%2Fforum%2F1219ec2d-6c26-ec11-b6e6-000d3a4f032c%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3EData%20factory%20%C2%B7%20Community%20(azure.com)%3C%2FA%3E%3CSPAN%3E.%3C%2FSPAN%3E%3C%2FSTRONG%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-3298972%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Noelle_Li_5-1651251208412.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F368051i92646264C8FA2376%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22Noelle_Li_5-1651251208412.png%22%20alt%3D%22Noelle_Li_5-1651251208412.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-3298972%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Data%20Integration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMapping%20Data%20Flows%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎May 06 2022 07:41 PM
Updated by: