I am trying to an ETL related activity using Data factory.
Currently, I have a copy activity copying data from CRM via the Common Data Service connector, to a CSV file.
Then I load the data in the file via a wrangling data flow activity to perform transformations. At the moment, I am deleting and renaming a number of columns to test the pipeline end to end.
The result after the transformation is now saved in a table in Azure synapse.
The issue I am having:
For some odd reason, I am experiencing loss of data in the database (Azure synapse) after performing the transformation.
For example: In the .csv file, all through to the wrangling data flow preview. I can see the rows fully populated. But in the database, I notice some rows have null in the first name, but the last name is populated. The row below will be missing the last name, but have the first name populated etc. Just random loss of data.
I have tried searching online for a solution, can't seem to get any.
Any ideas?
Thanks in advance.
