Ingest and Transform Data with Azure Synapse Analytics With Ease

Published 12-14-2020 09:00 AM 6,121 Views
Microsoft

The Integrate Hub within Azure Synapse Analytics workspace helps to manage, creating data integration pipelines for data movement and transformation. In this article, we will use the knowledge center inside the Synapse Studio to immediately ingest some sample data, US population by gender and race for each US ZIP code sourced from 2010 Decennial Census. Once we have access to the data, we will use a data flow to filter out what we don’t need and transform the data to create a final CSV file stored in the Azure Data Lake Storage Gen2.

 

Using Sample Data From Azure Synapse Knowledge Center

Our first step will be to get access to the data we need. Inside the Synapse workspace, choose the Data option from the left menu to open the Data Hub.

Shirley_Wang_0-1607598567224.png

Data Hub is open. The plus button to add new artifacts is selected. Browser Gallery from the list of linked data source options is highlighted.

 

Once in the gallery, make sure Datasets page is selected. From the list of available sample datasets, choose US Population by ZIP Code and select Continue.

Shirley_Wang_1-1607598567243.png

Knowledge Center dataset gallery is open. US Population by ZIP Code dataset is selected. Continue button is highlighted.

 

On the next screen, you can see the dataset’s details and observe a preview of the data. Select Add dataset to include the dataset into your Synapse workspace as a linked service.

Shirley_Wang_2-1607598567255.png

US Population by ZIP Code dataset details are shown. Add dataset button is highlighted.

 

Data Exploration With the Serverless SQL Pool

Once the external dataset is included to your workspace as a linked service, it will show up under the Data hub as under Azure Blog Storage / Sample Datasets section. Select the Actions button for us-decennial-census-zip represented as “…” to see a list of available actions for the linked data source.

Shirley_Wang_3-1607598567259.png

us-decennial-census-zip sample dataset is selected. The actions button is highlighted.

 

We will now query the remote data source using our serverless SQL pool to test our access to the data source. From the Actions menu Select New SQL Script / Select Top 100 rows. This will bring an auto-generated SQL script.

Shirley_Wang_4-1607598567264.png

The actions menu for us-decennial-census-zip is shown. New SQL script is selected, and Select Top 100 rows command is highlighted.

 

Select Run and see the data presented in Results tab.

Shirley_Wang_5-1607598567277.png

The actions menu for us-decennial-census-zip is shown. New SQL script is selected, and Select Top 100 rows command is highlighted.

 

Creating Integration Datasets for Ingest and Egress

Now that we proved our access to the data source, we can create integration datasets that will be our source and sink datasets. Feel free to close the current SQL Script window and discard all changes.

Shirley_Wang_6-1607598567282.png

The Data hub is on screen. Add new resource button is selected. From the list of resources integration dataset is highlighted.

 

Select the “+” Add new resource button to open the list of available resources we can add to our Synapse workspace. We will create two integration datasets for our example. The first one will help us integrate our newly added remote data source into a data flow. The second one will be an integration point for the sink CSV file we will create in our Azure Data Lake Storage Gen2 location.

Select Integration dataset to see a list of all data sources you can include. From the list, pick Azure Blob Storage and select Continue.

Shirley_Wang_7-1607598567290.png

New integration dataset window is open. Azure Blob Storage option is selected. Continue button is highlighted.

 

Our source files are in the Parquet format. Pick Parquet as the file format type and select Continue.

Shirley_Wang_8-1607598567294.png

File format options window is open. Parquet is selected. Continue button is highlighted.

 

On the next screen set the dataset name to CensusParquetDataset. Select us-decennial-census-zip as the Linked service. Finally, for the file path set censusdatacontainer for Container, and release/us_population_zip/year=2010 for Directory. Select OK to continue.

Shirley_Wang_9-1607598567298.png

Dataset properties window is open. The dataset name is set to CensusParquetDataset. Linked Service is set to us-decennial-census-zip. File path is set to censusdatacontainer. File Path folder is set to release/us_population_zip/year=2010. Finally, the OK button is highlighted.

 

Select the “+” Add new resource button and pick Integration dataset again to create our sink dataset. For this round, choose Azure Data Lake Storage Gen2 for the data store.

Shirley_Wang_10-1607598567305.png

New integration dataset window is open. Azure Data Lake Storage Gen2 option is selected. Continue button is highlighted.

 

Our target file will be a CSV file. Pick DelimitedText as the file format type and select Continue.

Shirley_Wang_11-1607598567309.png

File format options window is open. DelimitedText is selected. Continue button is highlighted.

 

On the next screen, set the dataset name to USCensus. Select Synapse workspace default storage as the Linked service. Finally, for the file path, select the Browse button and pick a location to save the CSV file. Select OK to continue. In our case, we selected default-fs as the file system and will be saving the file to the root folder.

Shirley_Wang_12-1607598567313.png

Dataset properties window is open. The dataset name is set to USCensus. Linked Service is set to the default workspace storage. File path is set to default-fs. Finally, the OK button is highlighted.

 

Now that we have both our source and sink datasets ready. It is time to select Publish all and save all the changes we implemented.

Shirley_Wang_13-1607598567317.png

CensusParquetDataset and USCensus datasets are open. Publish All button is highlighted.

 

Data Transformation With a Pipeline and a Dataflow

Switch to the Integrate Hub from the left menu. Select the “+” Add new resource button and select Pipeline to create a new Synapse Pipeline.

Shirley_Wang_14-1607598567321.png

Integrate Hub is open. Add Resource is selected. Pipeline command is highlighted.

 

Name the new pipeline USCensusPipeline and search for data in the Activities panel. Select Data flow activity and drag and drop one onto the screen.

Shirley_Wang_15-1607598567328.png

New pipeline window is open. Pipeline’s name is set to USCensusPipeline. The activities search box is searched for data. Data flow activity is highlighted.

 

Pick Create new data flow and select Data flow. Select Continue.

Shirley_Wang_16-1607598567331.png

New data flow window is open. Create new data flow option is selected. Data flow is selected. The OK button is highlighted.

 

Once you are in the Dataflow designer give your Dataflow a name. In our case, it is USCensusDataFlow. Select CensusParquetDataset as the Dataset in the Source settings window.

Shirley_Wang_17-1607598567338.png

Dataflow name is set to USCensusDataFlow. The source dataset is set to CensusParquetDataset.

 

Select the “+” button on the right of the source in the designer panel. This will open the list of transformation you can do. We will filter out the data to only include the ZIP Code 98101. Select Filter Row Modifier to add the activitiy into the data flow.

Shirley_Wang_18-1607598567343.png

Data transformations menu is open. Filter row modifier is highlighted.

 

On the Filter settings tab you can define your filtering expression in the Filter on field. Select the field box to navigate to the Visual expression builder.

Shirley_Wang_19-1607598567347.png

Filter settings tab is open. Filter on field is highlighted.

 

The data flow expression builder lets you interactively build expressions to use in various transformations. In our case, we will use equals(zipCode, "98101") to make sure we only return records with the 98101 ZIP code. Before selecting Save and finish make sure you take a look at the Expression elements list and the Expression values that includes the fields extracted from the source data schema.

Shirley_Wang_20-1607598567352.png

Visual expression builder is open. Expression is set to equals(zipCode, “98101”). Expression elements, expression values are highlighted. Save and finish button is selected.

 

Our data flow’s final step is to create the CSV file in Azure Data Lake Storage Gen2 for further use. We will add another step to our data flow. This time, it will be a sink as our output destination.

Shirley_Wang_21-1607598567357.png

Data transformations menu is open. Sink is highlighted.

 

Once the sink is in place, set the dataset to USCensus. This is the dataset targeting our Azure Data Lake Storage Gen2.

Shirley_Wang_22-1607598567362.png

Sink dataset is set to USCensus.

 

Switch to the Settings tab for the sink. Set the File name option to Output to single file. Output to single file combines all the data into a single partition. This leads to long write times, especially for large datasets. In our case, the final dataset will be very small. Finally, set the Output to single file value to USCensus.csv.

Shirley_Wang_23-1607598567368.png

Sink settings panel is open. Output to single file is selected and it’s value is set to USCensus.csv. Set Single partition button is highlighted.

 

Now, it’s time to hit Publish All and save all the changes.

Shirley_Wang_24-1607598567370.png

Publish all button is highlighted.

 

Close the data flow editor. Go back to our USCensusPipeline. Select Add Trigger and Trigger now to kick start the pipeline for the first time.

Shirley_Wang_25-1607598567376.png

USCensusPipeline is open. Add trigger is selected and Trigger now command is highlighted.

 

Results Are In

To monitor the pipeline execution go to the Monitor Hub. Select Pipeline runs and observe your pipeline executing.

Shirley_Wang_26-1607598567384.png

Monitor Hub is open. Pipeline runs is selected. USCensusPipeline pipeline row is highlighted showing status as in progress.

 

Once the pipeline is successfully executed, you can go back to the Data hub to see the final result. Select the workspace’s default Azure Data Lake Storage Gen2 location that you selected as your sink and find the resulting USCensus.csv file that is only 26.4KB with census data for the ZIP code used as the filter.

Shirley_Wang_27-1607598567390.png

Data Hub is open. Workspace’s default ADLS Gen2 storage is open. USCensus.csv file and its 26.5KB size is highlighted.

 

Keeping Things Clean and Tidy

We created a list of artifacts in our Synapse workspace. Let’s do some cleaning before we go.
First of all, delete the CSV file our pipeline created. Go to the Data hub, select the workspace’s default Azure Data Lake Storage Gen2 location that you selected as your sink and find the resulting USCensus.csv file. Select Delete from the command bar to delete the file.

Shirley_Wang_28-1607598567408.png

Data Hub is open. Workspace’s default ADLS Gen2 storage is open. USCensus.csv file is selected. Delete command is highlighted.

 

Go to the Integrate hub and select the USCensusPipeline we created. Select the Actions command and select Delete to remove the pipeline. Don’t forget to hit Publish All to execute the delete operation on the Synapse workspace.

Shirley_Wang_29-1607598567413.png

Integrate Hub is open. USCensusPipeline is selected. Actions menu is open and delete command is highlighted.

 

Go to the Develop hub and select the USCensusDataFlow we created. Select the Actions command and select Delete to remove the dataflow. Don’t forget to hit Publish All to execute the delete operation on the Synapse workspace.

Shirley_Wang_30-1607598567418.png

Develop Hub is open. USCensusDataFlow is selected. Actions menu is open and delete command is highlighted.

 

Go to the Data hub and select the USCensus we created. Select the Actions command and select Delete to remove the dataset. Repeat the same steps for the CensusParquetDataset as well. Don’t forget to hit Publish All to execute the delete operation on the Synapse workspace.

Shirley_Wang_31-1607598567427.png

Develop Hub is open. USCensus is selected. Actions menu is open and delete command is highlighted.

 

Go to the Manage hub and select Linked services. From the list, select the us-decennial-census-zip. Select Delete to remove the linked service.

Shirley_Wang_32-1607598567435.png

Manage Hub is open. The Linked services section is selected. The delete button for us-decennial-census-zip linked service is highlighted.

 

We are done. Our environment is the same as it was before.

 

Conclusion

We connected to a third-party data storage service that has parquet files in it. We looked into the files with the serverless SQL Pool. We created a pipeline with a data flow connecting to the outside source, filtering data out and putting the result in a CSV file in our data lake. While going through these steps, we have met with linked services, datasets, pipelines, dataflows, and a beautiful expression builder.

 

Go ahead, try out this tutorial yourself today by creating an Azure Synapse workspace.

%3CLINGO-SUB%20id%3D%22lingo-sub-1975563%22%20slang%3D%22en-US%22%3EIngest%20and%20Transform%20Data%20with%20Azure%20Synapse%20Analytics%20With%20Ease%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1975563%22%20slang%3D%22en-US%22%3E%3CP%3EThe%20Integrate%20Hub%20within%20Azure%20Synapse%20Analytics%20workspace%20helps%20to%20manage%2C%20creating%20data%20integration%20pipelines%20for%20data%20movement%20and%20transformation.%20In%20this%20article%2C%20we%20will%20use%20the%20knowledge%20center%20inside%20the%20Synapse%20Studio%20to%20immediately%20ingest%20some%20sample%20data%2C%20%3CA%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fservices%2Fopen-datasets%2Fcatalog%2Fus-decennial-census-zip%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EUS%20population%20by%20gender%20and%20race%20for%20each%20US%20ZIP%20code%20sourced%20from%202010%20Decennial%20Census%3C%2FA%3E.%20Once%20we%20have%20access%20to%20the%20data%2C%20we%20will%20use%20a%20data%20flow%20to%20filter%20out%20what%20we%20don%E2%80%99t%20need%20and%20transform%20the%20data%20to%20create%20a%20final%20CSV%20file%20stored%20in%20the%20Azure%20Data%20Lake%20Storage%20Gen2.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--1180814543%22%20id%3D%22toc-hId--1180814580%22%3EUsing%20Sample%20Data%20From%20Azure%20Synapse%20Knowledge%20Center%3C%2FH2%3E%0A%3CP%3EOur%20first%20step%20will%20be%20to%20get%20access%20to%20the%20data%20we%20need.%20Inside%20the%20Synapse%20workspace%2C%20choose%20the%20%3CSTRONG%3EData%3C%2FSTRONG%3E%20option%20from%20the%20left%20menu%20to%20open%20the%20%3CSTRONG%3EData%20Hub%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_0-1607598567224.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239510iAC798C9947AE3A83%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_0-1607598567224.png%22%20alt%3D%22Shirley_Wang_0-1607598567224.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EData%20Hub%20is%20open.%20The%20plus%20button%20to%20add%20new%20artifacts%20is%20selected.%20Browser%20Gallery%20from%20the%20list%20of%20linked%20data%20source%20options%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20in%20the%20gallery%2C%20make%20sure%20%3CSTRONG%3EDatasets%3C%2FSTRONG%3E%20page%20is%20selected.%20From%20the%20list%20of%20available%20sample%20datasets%2C%20choose%20%3CSTRONG%3EUS%20Population%20by%20ZIP%20Code%3C%2FSTRONG%3E%20and%20select%20%3CSTRONG%3EContinue%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_1-1607598567243.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239512iAF0EFB2951592A5F%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_1-1607598567243.png%22%20alt%3D%22Shirley_Wang_1-1607598567243.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EKnowledge%20Center%20dataset%20gallery%20is%20open.%20US%20Population%20by%20ZIP%20Code%20dataset%20is%20selected.%20Continue%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOn%20the%20next%20screen%2C%20you%20can%20see%20the%20dataset%E2%80%99s%20details%20and%20observe%20a%20preview%20of%20the%20data.%20Select%20%3CSTRONG%3EAdd%20dataset%3C%2FSTRONG%3E%20to%20include%20the%20dataset%20into%20your%20Synapse%20workspace%20as%20a%20linked%20service.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_2-1607598567255.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239511i1273CEAB4EBFB0A7%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_2-1607598567255.png%22%20alt%3D%22Shirley_Wang_2-1607598567255.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EUS%20Population%20by%20ZIP%20Code%20dataset%20details%20are%20shown.%20Add%20dataset%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1306698290%22%20id%3D%22toc-hId-1306698253%22%3EData%20Exploration%20With%20the%20Serverless%20SQL%20Pool%3C%2FH2%3E%0A%3CP%3EOnce%20the%20external%20dataset%20is%20included%20to%20your%20workspace%20as%20a%20linked%20service%2C%20it%20will%20show%20up%20under%20the%20%3CSTRONG%3EData%3C%2FSTRONG%3E%20hub%20as%20under%20%3CSTRONG%3EAzure%20Blog%20Storage%3C%2FSTRONG%3E%20%2F%20%3CSTRONG%3ESample%20Datasets%3C%2FSTRONG%3E%20section.%20Select%20the%20%3CSTRONG%3EActions%3C%2FSTRONG%3E%20button%20for%20%3CSTRONG%3Eus-decennial-census-zip%3C%2FSTRONG%3E%20represented%20as%20%E2%80%9C%E2%80%A6%E2%80%9D%20to%20see%20a%20list%20of%20available%20actions%20for%20the%20linked%20data%20source.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_3-1607598567259.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239513i65F3916505AB4EA6%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_3-1607598567259.png%22%20alt%3D%22Shirley_Wang_3-1607598567259.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3Eus-decennial-census-zip%20sample%20dataset%20is%20selected.%20The%20actions%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20will%20now%20query%20the%20remote%20data%20source%20using%20our%20serverless%20SQL%20pool%20to%20test%20our%20access%20to%20the%20data%20source.%20From%20the%20%3CSTRONG%3EActions%3C%2FSTRONG%3E%20menu%20Select%20%3CSTRONG%3ENew%20SQL%20Script%20%2F%20Select%20Top%20100%20rows%3C%2FSTRONG%3E.%20This%20will%20bring%20an%20auto-generated%20SQL%20script.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_4-1607598567264.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239514i10461B88BF8B2115%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_4-1607598567264.png%22%20alt%3D%22Shirley_Wang_4-1607598567264.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EThe%20actions%20menu%20for%20us-decennial-census-zip%20is%20shown.%20New%20SQL%20script%20is%20selected%2C%20and%20Select%20Top%20100%20rows%20command%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESelect%20%3CSTRONG%3ERun%3C%2FSTRONG%3E%20and%20see%20the%20data%20presented%20in%20%3CSTRONG%3EResults%3C%2FSTRONG%3E%20tab.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_5-1607598567277.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239515iC1E5F9EB51151088%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_5-1607598567277.png%22%20alt%3D%22Shirley_Wang_5-1607598567277.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EThe%20actions%20menu%20for%20us-decennial-census-zip%20is%20shown.%20New%20SQL%20script%20is%20selected%2C%20and%20Select%20Top%20100%20rows%20command%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--500756173%22%20id%3D%22toc-hId--500756210%22%3ECreating%20Integration%20Datasets%20for%20Ingest%20and%20Egress%3C%2FH2%3E%0A%3CP%3ENow%20that%20we%20proved%20our%20access%20to%20the%20data%20source%2C%20we%20can%20create%20integration%20datasets%20that%20will%20be%20our%20source%20and%20sink%20datasets.%20Feel%20free%20to%20close%20the%20current%20SQL%20Script%20window%20and%20discard%20all%20changes.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_6-1607598567282.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239517i7040F6CA58CFBB1E%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_6-1607598567282.png%22%20alt%3D%22Shirley_Wang_6-1607598567282.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EThe%20Data%20hub%20is%20on%20screen.%20Add%20new%20resource%20button%20is%20selected.%20From%20the%20list%20of%20resources%20integration%20dataset%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESelect%20the%20%3CSTRONG%3E%E2%80%9C%2B%E2%80%9D%20Add%20new%20resource%3C%2FSTRONG%3E%20button%20to%20open%20the%20list%20of%20available%20resources%20we%20can%20add%20to%20our%20Synapse%20workspace.%20We%20will%20create%20two%20integration%20datasets%20for%20our%20example.%20The%20first%20one%20will%20help%20us%20integrate%20our%20newly%20added%20remote%20data%20source%20into%20a%20data%20flow.%20The%20second%20one%20will%20be%20an%20integration%20point%20for%20the%20sink%20CSV%20file%20we%20will%20create%20in%20our%20Azure%20Data%20Lake%20Storage%20Gen2%20location.%3C%2FP%3E%0A%3CP%3ESelect%20%3CSTRONG%3EIntegration%20dataset%3C%2FSTRONG%3E%20to%20see%20a%20list%20of%20all%20data%20sources%20you%20can%20include.%20From%20the%20list%2C%20pick%20%3CSTRONG%3EAzure%20Blob%20Storage%3C%2FSTRONG%3E%20and%20select%20%3CSTRONG%3EContinue%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_7-1607598567290.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239516iD3B527CC4ACCF6BC%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_7-1607598567290.png%22%20alt%3D%22Shirley_Wang_7-1607598567290.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ENew%20integration%20dataset%20window%20is%20open.%20Azure%20Blob%20Storage%20option%20is%20selected.%20Continue%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOur%20source%20files%20are%20in%20the%20Parquet%20format.%20Pick%20%3CSTRONG%3EParquet%3C%2FSTRONG%3E%20as%20the%20file%20format%20type%20and%20select%20%3CSTRONG%3EContinue%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_8-1607598567294.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239518iBF3426C26D3BEFAA%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_8-1607598567294.png%22%20alt%3D%22Shirley_Wang_8-1607598567294.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EFile%20format%20options%20window%20is%20open.%20Parquet%20is%20selected.%20Continue%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOn%20the%20next%20screen%20set%20the%20dataset%20%3CSTRONG%3Ename%3C%2FSTRONG%3E%20to%20%3CSPAN%3ECensusParquetDataset%3C%2FSPAN%3E.%20Select%20%3CSPAN%3Eus-decennial-census-zip%3C%2FSPAN%3E%20as%20the%20%3CSTRONG%3ELinked%20service%3C%2FSTRONG%3E.%20Finally%2C%20for%20the%20file%20path%20set%20%3CSPAN%3Ecensusdatacontainer%3C%2FSPAN%3E%20for%20Container%2C%20and%20%3CSPAN%3Erelease%2Fus_population_zip%2Fyear%3D2010%3C%2FSPAN%3E%20for%20Directory.%20Select%20%3CSTRONG%3EOK%3C%2FSTRONG%3E%20to%20continue.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_9-1607598567298.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239519i84326F0ADE510E0B%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_9-1607598567298.png%22%20alt%3D%22Shirley_Wang_9-1607598567298.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EDataset%20properties%20window%20is%20open.%20The%20dataset%20name%20is%20set%20to%20CensusParquetDataset.%20Linked%20Service%20is%20set%20to%20us-decennial-census-zip.%20File%20path%20is%20set%20to%20censusdatacontainer.%20File%20Path%20folder%20is%20set%20to%20release%2Fus_population_zip%2Fyear%3D2010.%20Finally%2C%20the%20OK%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESelect%20the%20%3CSTRONG%3E%E2%80%9C%2B%E2%80%9D%20Add%20new%20resource%3C%2FSTRONG%3E%20button%20and%20pick%20%3CSTRONG%3EIntegration%20dataset%3C%2FSTRONG%3E%20again%20to%20create%20our%20sink%20dataset.%20For%20this%20round%2C%20choose%20%3CSTRONG%3EAzure%20Data%20Lake%20Storage%20Gen2%3C%2FSTRONG%3E%20for%20the%20data%20store.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_10-1607598567305.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239520i8AB0FCCE12E6B4C1%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_10-1607598567305.png%22%20alt%3D%22Shirley_Wang_10-1607598567305.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ENew%20integration%20dataset%20window%20is%20open.%20Azure%20Data%20Lake%20Storage%20Gen2%20option%20is%20selected.%20Continue%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOur%20target%20file%20will%20be%20a%20CSV%20file.%20Pick%20%3CSTRONG%3EDelimitedText%3C%2FSTRONG%3E%20as%20the%20file%20format%20type%20and%20select%20%3CSTRONG%3EContinue%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_11-1607598567309.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239521i10C4BB3572210124%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_11-1607598567309.png%22%20alt%3D%22Shirley_Wang_11-1607598567309.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EFile%20format%20options%20window%20is%20open.%20DelimitedText%20is%20selected.%20Continue%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOn%20the%20next%20screen%2C%20set%20the%20dataset%20%3CSTRONG%3Ename%3C%2FSTRONG%3E%20to%20%3CSPAN%3EUSCensus%3C%2FSPAN%3E.%20Select%20Synapse%20workspace%20default%20storage%20as%20the%20%3CSTRONG%3ELinked%20service%3C%2FSTRONG%3E.%20Finally%2C%20for%20the%20file%20path%2C%20select%20the%20%3CSTRONG%3EBrowse%3C%2FSTRONG%3E%20button%20and%20pick%20a%20location%20to%20save%20the%20CSV%20file.%20Select%20%3CSTRONG%3EOK%3C%2FSTRONG%3E%20to%20continue.%20In%20our%20case%2C%20we%20selected%20%3CSPAN%3Edefault-fs%3C%2FSPAN%3E%20as%20the%20file%20system%20and%20will%20be%20saving%20the%20file%20to%20the%20root%20folder.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_12-1607598567313.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239522i7EFEE1B212D4A0B5%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_12-1607598567313.png%22%20alt%3D%22Shirley_Wang_12-1607598567313.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EDataset%20properties%20window%20is%20open.%20The%20dataset%20name%20is%20set%20to%20USCensus.%20Linked%20Service%20is%20set%20to%20the%20default%20workspace%20storage.%20File%20path%20is%20set%20to%20default-fs.%20Finally%2C%20the%20OK%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENow%20that%20we%20have%20both%20our%20source%20and%20sink%20datasets%20ready.%20It%20is%20time%20to%20select%20%3CSTRONG%3EPublish%20all%3C%2FSTRONG%3E%20and%20save%20all%20the%20changes%20we%20implemented.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_13-1607598567317.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239524i2ED48E1A18915D6D%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_13-1607598567317.png%22%20alt%3D%22Shirley_Wang_13-1607598567317.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ECensusParquetDataset%20and%20USCensus%20datasets%20are%20open.%20Publish%20All%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1986756660%22%20id%3D%22toc-hId-1986756623%22%3EData%20Transformation%20With%20a%20Pipeline%20and%20a%20Dataflow%3C%2FH2%3E%0A%3CP%3ESwitch%20to%20the%20%3CSTRONG%3EIntegrate%3C%2FSTRONG%3E%20Hub%20from%20the%20left%20menu.%20Select%20the%20%3CSTRONG%3E%E2%80%9C%2B%E2%80%9D%20Add%20new%20resource%3C%2FSTRONG%3E%20button%20and%20select%20%3CSTRONG%3EPipeline%3C%2FSTRONG%3E%20to%20create%20a%20new%20Synapse%20Pipeline.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_14-1607598567321.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239523i899EB6B43BC1DA27%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_14-1607598567321.png%22%20alt%3D%22Shirley_Wang_14-1607598567321.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EIntegrate%20Hub%20is%20open.%20Add%20Resource%20is%20selected.%20Pipeline%20command%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EName%20the%20new%20pipeline%20%3CSPAN%3EUSCensusPipeline%3C%2FSPAN%3E%20and%20search%20for%20%3CSPAN%3Edata%3C%2FSPAN%3E%20in%20the%20%3CSTRONG%3EActivities%3C%2FSTRONG%3E%20panel.%20Select%20%3CSTRONG%3EData%20flow%3C%2FSTRONG%3E%20activity%20and%20drag%20and%20drop%20one%20onto%20the%20screen.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_15-1607598567328.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239525iB651FBF7E6FF6513%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_15-1607598567328.png%22%20alt%3D%22Shirley_Wang_15-1607598567328.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ENew%20pipeline%20window%20is%20open.%20Pipeline%E2%80%99s%20name%20is%20set%20to%20USCensusPipeline.%20The%20activities%20search%20box%20is%20searched%20for%20data.%20Data%20flow%20activity%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EPick%20%3CSTRONG%3ECreate%20new%20data%20flow%3C%2FSTRONG%3E%20and%20select%20%3CSTRONG%3EData%20flow%3C%2FSTRONG%3E.%20Select%20%3CSTRONG%3EContinue%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_16-1607598567331.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239526i6DDA7BCD35A209B2%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_16-1607598567331.png%22%20alt%3D%22Shirley_Wang_16-1607598567331.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ENew%20data%20flow%20window%20is%20open.%20Create%20new%20data%20flow%20option%20is%20selected.%20Data%20flow%20is%20selected.%20The%20OK%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20you%20are%20in%20the%20%3CSTRONG%3EDataflow%3C%2FSTRONG%3E%20designer%20give%20your%20Dataflow%20a%20name.%20In%20our%20case%2C%20it%20is%20%3CSPAN%3EUSCensusDataFlow%3C%2FSPAN%3E.%20Select%20%3CSPAN%3ECensusParquetDataset%3C%2FSPAN%3E%20as%20the%20%3CSTRONG%3EDataset%3C%2FSTRONG%3E%20in%20the%20%3CSTRONG%3ESource%20settings%3C%2FSTRONG%3E%20window.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_17-1607598567338.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239527i3A30D14604C8FDA0%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_17-1607598567338.png%22%20alt%3D%22Shirley_Wang_17-1607598567338.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EDataflow%20name%20is%20set%20to%20USCensusDataFlow.%20The%20source%20dataset%20is%20set%20to%20CensusParquetDataset.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESelect%20the%20%E2%80%9C%2B%E2%80%9D%20button%20on%20the%20right%20of%20the%20source%20in%20the%20designer%20panel.%20This%20will%20open%20the%20list%20of%20transformation%20you%20can%20do.%20We%20will%20filter%20out%20the%20data%20to%20only%20include%20the%20ZIP%20Code%2098101.%20Select%20%3CSTRONG%3EFilter%3C%2FSTRONG%3E%20Row%20Modifier%20to%20add%20the%20activitiy%20into%20the%20data%20flow.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_18-1607598567343.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239529iC85A07776A5A57E3%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_18-1607598567343.png%22%20alt%3D%22Shirley_Wang_18-1607598567343.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EData%20transformations%20menu%20is%20open.%20Filter%20row%20modifier%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOn%20the%20%3CSTRONG%3EFilter%20settings%3C%2FSTRONG%3E%20tab%20you%20can%20define%20your%20filtering%20expression%20in%20the%20%3CSTRONG%3EFilter%20on%3C%2FSTRONG%3E%20field.%20Select%20the%20field%20box%20to%20navigate%20to%20the%20%3CSTRONG%3EVisual%20expression%20builder%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_19-1607598567347.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239528i6A2BE6D93CA98165%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_19-1607598567347.png%22%20alt%3D%22Shirley_Wang_19-1607598567347.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EFilter%20settings%20tab%20is%20open.%20Filter%20on%20field%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20data%20flow%20expression%20builder%20lets%20you%20interactively%20build%20expressions%20to%20use%20in%20various%20transformations.%20In%20our%20case%2C%20we%20will%20use%20%3CSPAN%3Eequals(zipCode%2C%20%2298101%22)%3C%2FSPAN%3E%20to%20make%20sure%20we%20only%20return%20records%20with%20the%2098101%20ZIP%20code.%20Before%20selecting%20%3CSTRONG%3ESave%20and%20finish%3C%2FSTRONG%3E%20make%20sure%20you%20take%20a%20look%20at%20the%20%3CSTRONG%3EExpression%20elements%3C%2FSTRONG%3E%20list%20and%20the%20%3CSTRONG%3EExpression%20values%3C%2FSTRONG%3E%20that%20includes%20the%20fields%20extracted%20from%20the%20source%20data%20schema.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_20-1607598567352.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239530i52E7EF807B70FE20%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_20-1607598567352.png%22%20alt%3D%22Shirley_Wang_20-1607598567352.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EVisual%20expression%20builder%20is%20open.%20Expression%20is%20set%20to%20equals(zipCode%2C%20%E2%80%9C98101%E2%80%9D).%20Expression%20elements%2C%20expression%20values%20are%20highlighted.%20Save%20and%20finish%20button%20is%20selected.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOur%20data%20flow%E2%80%99s%20final%20step%20is%20to%20create%20the%20CSV%20file%20in%20Azure%20Data%20Lake%20Storage%20Gen2%20for%20further%20use.%20We%20will%20add%20another%20step%20to%20our%20data%20flow.%20This%20time%2C%20it%20will%20be%20a%20sink%20as%20our%20output%20destination.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_21-1607598567357.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239531iF2288D15AB085237%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_21-1607598567357.png%22%20alt%3D%22Shirley_Wang_21-1607598567357.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EData%20transformations%20menu%20is%20open.%20Sink%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20the%20%3CSTRONG%3Esink%3C%2FSTRONG%3E%20is%20in%20place%2C%20set%20the%20%3CSTRONG%3Edataset%3C%2FSTRONG%3E%20to%20%3CSPAN%3EUSCensus%3C%2FSPAN%3E.%20This%20is%20the%20dataset%20targeting%20our%20Azure%20Data%20Lake%20Storage%20Gen2.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_22-1607598567362.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239532i80E65CEBA46C766F%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_22-1607598567362.png%22%20alt%3D%22Shirley_Wang_22-1607598567362.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ESink%20dataset%20is%20set%20to%20USCensus.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESwitch%20to%20the%20%3CSTRONG%3ESettings%3C%2FSTRONG%3E%20tab%20for%20the%20sink.%20Set%20the%20%3CSTRONG%3EFile%20name%20option%3C%2FSTRONG%3E%20to%20%3CSTRONG%3EOutput%20to%20single%20file%3C%2FSTRONG%3E.%20%3CSTRONG%3EOutput%20to%20single%20file%3C%2FSTRONG%3E%20combines%20all%20the%20data%20into%20a%20single%20partition.%20This%20leads%20to%20long%20write%20times%2C%20especially%20for%20large%20datasets.%20In%20our%20case%2C%20the%20final%20dataset%20will%20be%20very%20small.%20Finally%2C%20set%20the%20%3CSTRONG%3EOutput%20to%20single%20file%3C%2FSTRONG%3E%20value%20to%20%3CSPAN%3EUSCensus.csv%3C%2FSPAN%3E.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_23-1607598567368.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239533iE4A232575714A562%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_23-1607598567368.png%22%20alt%3D%22Shirley_Wang_23-1607598567368.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3ESink%20settings%20panel%20is%20open.%20Output%20to%20single%20file%20is%20selected%20and%20it%E2%80%99s%20value%20is%20set%20to%20USCensus.csv.%20Set%20Single%20partition%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENow%2C%20it%E2%80%99s%20time%20to%20hit%20%3CSTRONG%3EPublish%20All%3C%2FSTRONG%3E%20and%20save%20all%20the%20changes.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_24-1607598567370.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239534iC28A3D56371AC0A5%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_24-1607598567370.png%22%20alt%3D%22Shirley_Wang_24-1607598567370.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EPublish%20all%20button%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EClose%20the%20data%20flow%20editor.%20Go%20back%20to%20our%20%3CSPAN%3EUSCensusPipeline%3C%2FSPAN%3E.%20Select%20%3CSTRONG%3EAdd%20Trigger%3C%2FSTRONG%3E%20and%20%3CSTRONG%3ETrigger%20now%3C%2FSTRONG%3E%20to%20kick%20start%20the%20pipeline%20for%20the%20first%20time.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_25-1607598567376.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239535iB8A45C03613AFFD7%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_25-1607598567376.png%22%20alt%3D%22Shirley_Wang_25-1607598567376.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EUSCensusPipeline%20is%20open.%20Add%20trigger%20is%20selected%20and%20Trigger%20now%20command%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-179302197%22%20id%3D%22toc-hId-179302160%22%3EResults%20Are%20In%3C%2FH2%3E%0A%3CP%3ETo%20monitor%20the%20pipeline%20execution%20go%20to%20the%20%3CSTRONG%3EMonitor%3C%2FSTRONG%3E%20Hub.%20Select%20%3CSTRONG%3EPipeline%20runs%3C%2FSTRONG%3E%20and%20observe%20your%20pipeline%20executing.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_26-1607598567384.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239536i7038F67DED6AFE6E%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_26-1607598567384.png%22%20alt%3D%22Shirley_Wang_26-1607598567384.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EMonitor%20Hub%20is%20open.%20Pipeline%20runs%20is%20selected.%20USCensusPipeline%20pipeline%20row%20is%20highlighted%20showing%20status%20as%20in%20progress.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20the%20pipeline%20is%20successfully%20executed%2C%20you%20can%20go%20back%20to%20the%20%3CSTRONG%3EData%3C%2FSTRONG%3E%20hub%20to%20see%20the%20final%20result.%20Select%20the%20workspace%E2%80%99s%20default%20%3CSTRONG%3EAzure%20Data%20Lake%20Storage%20Gen2%3C%2FSTRONG%3E%20location%20that%20you%20selected%20as%20your%20sink%20and%20find%20the%20resulting%20%3CSPAN%3EUSCensus.csv%3C%2FSPAN%3E%20file%20that%20is%20only%2026.4KB%20with%20census%20data%20for%20the%20ZIP%20code%20used%20as%20the%20filter.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_27-1607598567390.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239537iFCAE5C2B0018561A%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_27-1607598567390.png%22%20alt%3D%22Shirley_Wang_27-1607598567390.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EData%20Hub%20is%20open.%20Workspace%E2%80%99s%20default%20ADLS%20Gen2%20storage%20is%20open.%20USCensus.csv%20file%20and%20its%2026.5KB%20size%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--1628152266%22%20id%3D%22toc-hId--1628152303%22%3EKeeping%20Things%20Clean%20and%20Tidy%3C%2FH2%3E%0A%3CP%3EWe%20created%20a%20list%20of%20artifacts%20in%20our%20Synapse%20workspace.%20Let%E2%80%99s%20do%20some%20cleaning%20before%20we%20go.%3CBR%20%2F%3EFirst%20of%20all%2C%20delete%20the%20CSV%20file%20our%20pipeline%20created.%20Go%20to%20the%20%3CSTRONG%3EData%3C%2FSTRONG%3E%20hub%2C%20select%20the%20workspace%E2%80%99s%20default%20%3CSTRONG%3EAzure%20Data%20Lake%20Storage%20Gen2%3C%2FSTRONG%3E%20location%20that%20you%20selected%20as%20your%20sink%20and%20find%20the%20resulting%20%3CSPAN%3EUSCensus.csv%3C%2FSPAN%3E%20file.%20Select%20%3CSTRONG%3EDelete%3C%2FSTRONG%3E%20from%20the%20command%20bar%20to%20delete%20the%20file.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_28-1607598567408.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239539i55A91AB943760295%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_28-1607598567408.png%22%20alt%3D%22Shirley_Wang_28-1607598567408.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EData%20Hub%20is%20open.%20Workspace%E2%80%99s%20default%20ADLS%20Gen2%20storage%20is%20open.%20USCensus.csv%20file%20is%20selected.%20Delete%20command%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EGo%20to%20the%20%3CSTRONG%3EIntegrate%3C%2FSTRONG%3E%20hub%20and%20select%20the%20%3CSPAN%3EUSCensusPipeline%3C%2FSPAN%3E%20we%20created.%20Select%20the%20%3CSTRONG%3EActions%3C%2FSTRONG%3E%20command%20and%20select%20%3CSTRONG%3EDelete%3C%2FSTRONG%3E%20to%20remove%20the%20pipeline.%20Don%E2%80%99t%20forget%20to%20hit%20%3CSTRONG%3EPublish%20All%3C%2FSTRONG%3E%20to%20execute%20the%20delete%20operation%20on%20the%20Synapse%20workspace.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_29-1607598567413.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239538iBB9646365134B491%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_29-1607598567413.png%22%20alt%3D%22Shirley_Wang_29-1607598567413.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EIntegrate%20Hub%20is%20open.%20USCensusPipeline%20is%20selected.%20Actions%20menu%20is%20open%20and%20delete%20command%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EGo%20to%20the%20%3CSTRONG%3EDevelop%3C%2FSTRONG%3E%20hub%20and%20select%20the%20%3CSPAN%3EUSCensusDataFlow%3C%2FSPAN%3E%20we%20created.%20Select%20the%20%3CSTRONG%3EActions%3C%2FSTRONG%3E%20command%20and%20select%20%3CSTRONG%3EDelete%3C%2FSTRONG%3E%20to%20remove%20the%20dataflow.%20Don%E2%80%99t%20forget%20to%20hit%20%3CSTRONG%3EPublish%20All%3C%2FSTRONG%3E%20to%20execute%20the%20delete%20operation%20on%20the%20Synapse%20workspace.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_30-1607598567418.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239540i7EC790BED3F39DEB%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_30-1607598567418.png%22%20alt%3D%22Shirley_Wang_30-1607598567418.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EDevelop%20Hub%20is%20open.%20USCensusDataFlow%20is%20selected.%20Actions%20menu%20is%20open%20and%20delete%20command%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EGo%20to%20the%20%3CSTRONG%3EData%3C%2FSTRONG%3E%20hub%20and%20select%20the%20%3CSPAN%3EUSCensus%3C%2FSPAN%3E%20we%20created.%20Select%20the%20%3CSTRONG%3EActions%3C%2FSTRONG%3E%20command%20and%20select%20%3CSTRONG%3EDelete%3C%2FSTRONG%3E%20to%20remove%20the%20dataset.%20Repeat%20the%20same%20steps%20for%20the%20%3CSPAN%3ECensusParquetDataset%3C%2FSPAN%3E%20as%20well.%20Don%E2%80%99t%20forget%20to%20hit%20%3CSTRONG%3EPublish%20All%3C%2FSTRONG%3E%20to%20execute%20the%20delete%20operation%20on%20the%20Synapse%20workspace.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_31-1607598567427.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239541i2FE48F6D00F847A1%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_31-1607598567427.png%22%20alt%3D%22Shirley_Wang_31-1607598567427.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EDevelop%20Hub%20is%20open.%20USCensus%20is%20selected.%20Actions%20menu%20is%20open%20and%20delete%20command%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EGo%20to%20the%20%3CSTRONG%3EManage%3C%2FSTRONG%3E%20hub%20and%20select%20%3CSTRONG%3ELinked%20services%3C%2FSTRONG%3E.%20From%20the%20list%2C%20select%20the%20%3CSPAN%3Eus-decennial-census-zip%3C%2FSPAN%3E.%20Select%20%3CSTRONG%3EDelete%3C%2FSTRONG%3E%20to%20remove%20the%20linked%20service.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Shirley_Wang_32-1607598567435.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239542i1DDFA8B697459958%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Shirley_Wang_32-1607598567435.png%22%20alt%3D%22Shirley_Wang_32-1607598567435.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3EManage%20Hub%20is%20open.%20The%20Linked%20services%20section%20is%20selected.%20The%20delete%20button%20for%20us-decennial-census-zip%20linked%20service%20is%20highlighted.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20are%20done.%20Our%20environment%20is%20the%20same%20as%20it%20was%20before.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-859360567%22%20id%3D%22toc-hId-859360530%22%3EConclusion%3C%2FH2%3E%0A%3CP%3EWe%20connected%20to%20a%20third-party%20data%20storage%20service%20that%20has%20parquet%20files%20in%20it.%20We%20looked%20into%20the%20files%20with%20the%20serverless%20SQL%20Pool.%20We%20created%20a%20pipeline%20with%20a%20data%20flow%20connecting%20to%20the%20outside%20source%2C%20filtering%20data%20out%20and%20putting%20the%20result%20in%20a%20CSV%20file%20in%20our%20data%20lake.%20While%20going%20through%20these%20steps%2C%20we%20have%20met%20with%20linked%20services%2C%20datasets%2C%20pipelines%2C%20dataflows%2C%20and%20a%20beautiful%20expression%20builder.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EGo%20ahead%2C%20try%20out%20this%20tutorial%20yourself%20today%20by%20creating%20an%20Azure%20Synapse%20workspace.%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Ffree%2Fsynapse-analytics%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ESign%20up%20for%20an%20Azure%20free%20account%3C%2FA%3E%20and%20receive%20%24200%20of%20credit%20to%20try%20Azure%20Synapse%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsynapse-analytics%2Fquickstart-create-workspace%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ECreate%20an%20Azure%20Synapse%20workspace%20in%20minutes%3C%2FA%3E%20to%20access%20these%20new%20features%3C%2FLI%3E%0A%3CLI%3EEmbark%20on%20your%20Azure%20Synapse%20self-study%20journey%20with%20the%26nbsp%3B%3CU%3E%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Faka.ms%252FAzureLearn_SynapseAnalytics%26amp%3Bdata%3D04%257C01%257Cqingyue.wang%2540microsoft.com%257C96765bce2f9f4f22d29b08d8a11bf759%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637436488768942766%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26amp%3Bsdata%3DAEUr%252BaWRFvLG7XasPjJCSbzhLcRQyGCQguTiRp5iHV8%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EMicrosoft%20Learn%20materials%3C%2FA%3E%3C%2FU%3E%26nbsp%3B%3C%2FLI%3E%0A%3CLI%3ECheck%20%3CU%3E%3CA%20href%3D%22https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Ftechcommunity.microsoft.com%252Ft5%252Fazure-synapse-analytics%252Fexplore-the-monitor-hub-in-synapse-studio-to-keep-track-of-all%252Fba-p%252Faka.ms%252Fsynapse-ga%26amp%3Bdata%3D04%257C01%257Cqingyue.wang%2540microsoft.com%257C96765bce2f9f4f22d29b08d8a11bf759%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637436488768942766%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26amp%3Bsdata%3DYvXB1E6XgCyaMR%252BtuqLyv5iJ1VYDuac9nYlDR2s1dC4%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ethis%20blog%20daily%20to%20see%20a%20roundup%20of%20all%20the%20new%20tutorial%20blogs%20that%3C%2FA%3E%3C%2FU%3E%20will%20be%20posted%20for%20the%20next%20two%20weeks%3C%2FLI%3E%0A%3C%2FUL%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1975563%22%20slang%3D%22en-US%22%3E%3CP%20class%3D%22FirstParagraph%22%3EThe%20Integrate%20Hub%20within%20Azure%20Synapse%20Analytics%20workspace%20helps%20to%20manage%2C%20creating%20data%20integration%20pipelines%20for%20data%20movement%20and%20transformation.%20In%20this%20article%2C%20we%20will%20use%20the%20knowledge%20center%20inside%20the%20Synapse%20Studio%20to%20ingest%20sample%20data%20and%20use%20a%20data%20flow%20to%20filter%20and%20transform%20the%20data%20to%20create%20a%20final%20CSV%20file%20in%20Azure%20Data%20Lake%20Storage%20Gen2.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1975563%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ESynapse%20Monitoring%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESynapse%20Pipelines%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESynapse%20Spark%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESynapse%20SQL%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESynapse%20Studio%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Version history
Last update:
‎Dec 15 2020 11:55 PM
Updated by: