Forum Discussion

sai_sathya's avatar
sai_sathya
Copper Contributor
Sep 29, 2023

validating record count at SQL server database tables with migrated azure data lake gen2

we are migrating out project from onpermise to azure , so on-premise database is the SQL server that we are using and azure datalake gen2 is the storage location where store data currently and so far we are currently validating record count of each table manually from sql server database tables and similarly we write pyspark code in databricks to write those data as parquet file and we validate record count from pyspark manually every time which is time consuming

 

is that possible execute this process to make it automated in order to save time ?

 

can this be done by using pyspark code ?

No RepliesBe the first to reply

Resources