MMLSpark provides a number of deep learning and data science tools for Apache Spark , including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK) and OpenCV , enabling you to quickly create powerful, highly-scalable predictive and analytical models for large image and text datasets.
DataFrame( example:301 )
See our notebooks for all examples.A short example
Below is an excerpt from a simple example of using a pre-trained CNN to classify images in the CIFAR-10 dataset. View the whole source code as an example notebook ....
The easiest way to evaluate MMLSpark is via our pre-built Docker container. To do so, run the following command:
docker run -it -p 8888:8888 -e ACCEPT_EULA=yes microsoft/mmlspark
To read the EULA for using the docker image, run docker run -it -p 8888:8888 microsoft/mmlspark eula
MMLSpark can be conveniently installed on existing Spark clusters via the
spark-shell --packages com.microsoft.ml.spark:mmlspark_2.11:0.6 \
pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:0.6 \
spark-submit --packages com.microsoft.ml.spark:mmlspark_2.11:0.6 \
--repositories https://mmlspark.azureedge.net/maven \
The script action url is: https://mmlspark.azureedge.net/buildartifacts/0.6/install-mmlspark.sh .
If you're using the Azure Portal to run the script action, go to
section of your cluster blade. In the
Bash script URI
field, input the script action URL provided above. Mark the rest of the options as shown on the screenshot to the right.
Submit, and the cluster should finish configuring within 10 minutes or so.Databricks cloud
For the coordinates use:
. Then, under Advanced Options, use
for the repository. Ensure this library is attached to all clusters you create.
Finally, ensure that your Spark cluster has at least Spark 2.1 and Scala 2.11.
You can use MMLSpark in both your Scala and PySpark notebooks.SBT
If you are building a Spark application in Scala, add the following lines to your
You can also easily create your own build by cloning this repo and use the main build script:
. Run it once to install the needed dependencies, and again to do a build. See
for more information and check out all the resources and documention at
Interested in learning more watch Joseph Sirosh keynote from PyData 2017 https://channel9.msdn.com/Events/PyData/Seattle2017/Key03
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.