Forum Discussion
Unable to configure log directory of Spark History Server to Storage Blob when deployed on AKS
I am trying to deploy Spark History Server on AKS and wanted to point it's log directory to Storage Blob. For achieving this, I am putting my configs on the values.yaml file as below:- wasbs:
e...
Not sure if you are still facing issue, you can add below jars to Dockerfile:
| # Add dependency for hadoop-azure |
ADD http://repo1.maven.org/maven2/com/microsoft/azure/azure-storage/2.0.0/azure-storage-2.0.0.jar $SPARK_HOME/jars
# Add hadoop-azure to access Azure Blob Storage
ADD http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure/2.7.3/hadoop-azure-2.7.3.jar $SPARK_HOME/jars
