MMLSpark provides a number of deep learning and data science tools for
, including seamless integration of Spark Machine Learning pipelines with
Microsoft Cognitive Toolkit (CNTK)
, enabling you to quickly create powerful, highly-scalable predictive and analytical models for large image and text datasets.
MMLSpark requires Scala 2.11, Spark 2.1+, and either Python 2.7 or Python 3.5+. See the API documentation
Easily ingest images from HDFS into Spark
Pre-process image data using transforms from OpenCV (
Featurize images using pre-trained deep neural nets using CNTK (
Train DNN-based image classification models on N-Series GPU VMs on Azure
Featurize free-form text data using convenient APIs on top of primitives in SparkML via a single transformer (
Train classification and regression models easily via implicit featurization of data (
Compute a rich set of evaluation metrics including per-instance metrics (
If you're using the Azure Portal to run the script action, go to
section of your cluster blade. In the
Bash script URI
field, input the script action URL provided above. Mark the rest of the options as shown on the screenshot to the right.
Submit, and the cluster should finish configuring within 10 minutes or so.
For the coordinates use:
. Then, under Advanced Options, use
for the repository. Ensure this library is attached to all clusters you create.
Finally, ensure that your Spark cluster has at least Spark 2.1 and Scala 2.11.
You can use MMLSpark in both your Scala and PySpark notebooks.
If you are building a Spark application in Scala, add the following lines to your
You can also easily create your own build by cloning this repo and use the main build script:
. Run it once to install the needed dependencies, and again to do a build. See
for more information and check out all the resources and documention at