Forum Discussion
Data Warehousing using Apache Spark on Azure HDinsight
Hi Team
Hope all are safe!
This is my first project in Azure and we are looking at developing a DW using Apache Spark on Azure HDinsight.
In simple terms we are currently trying to pick files from Share Point and then do transformations using pyspark and then load the data into a Azure Sql db.
Can someone help me on the below queries:
1) Can we connect Apache Spark or Pyspark on Azure HDinsight to Share Point to pick files?
2) Can we implement the usual SCD1 or SCD2 logic using pyspark?
Thanks in advance!
- BuckWoodyMSFT
Microsoft
Hello! I suggest you request support here - https://social.msdn.microsoft.com/Forums/sqlserver/en-US/home - Hi Aishwar04
This question is very old, but maybe this will help other people that will come here in the future as well.
Note! The link which you got in the MSDN forums is not active and it will redirect you to a general forum at MSDN named "Forums Issues (not product support)".
The English MSDN forums were migrated long time ago to a new system named Microsoft QnA which is part of the Microsoft Docs system. You can find it in this link: https://docs.microsoft.com/answers
The new system is based on tags and not separate forums. When you ask a question make sure that you are using a relevant tag. In your case you probably you can use the tag "azure-databricks"
> This is my first project in Azure
Congrats 🙂
> 1) Can we connect Apache Spark or Pyspark on Azure HDinsight to Share Point to pick files?
You can manage jobs in Azure Databricks using PySpark (Python API for Apache Spark).
I have not tried it but using the help of Google I found the following article which seems like bring the solution. Please check it:
https://www.cdata.com/kb/tech/sharepoint-jdbc-apache-spark.rst
I recommend to check the QnA forum as mentioned above for experts in the topic Yes, you can connect Apache Spark or PySpark on Azure HDInsight to SharePoint by using APIs or data connectors to access files, which can then be integrated into your data warehouse for further processing and analysis.
