%3CLINGO-SUB%20id%3D%22lingo-sub-1547960%22%20slang%3D%22en-US%22%3ETrain%20and%20Score%20Hundreds%20of%20Thousands%20of%20Models%20in%20Parallel%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1547960%22%20slang%3D%22en-US%22%3E%3CH2%20id%3D%22toc-hId--1298038261%22%20id%3D%22toc-hId--1298038261%22%20id%3D%22toc-hId--1298038261%22%3E%3CEM%3EAbstract%3C%2FEM%3E%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWith%20the%20-ERR%3AREF-NOT-FOUND-%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2F%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAzure%20Machine%20Learning%20service%3C%2FA%3E%2C%20the%20training%20and%20scoring%20of%20hundreds%20of%20thousands%20of%20models%20with%20large%20amounts%20of%20data%20can%20be%20completed%20efficiently%20leveraging%20pipelines%20where%20certain%20steps%20like%20model%20training%20and%20model%20scoring%20run%20in%20parallel%20on%20large%20scale%20out%20compute%20clusters.%20In%20order%20to%20help%20organizations%20get%20a%20head%20start%20on%20building%20such%20pipelines%2C%20the%20%5B%23%24dp7%5D%3CEM%3EMany%20Models%20Solution%20Accelerator%3C%2FEM%3E%20has%20been%20created.%20The%20Many%20Models%20Solution%20Accelerator%20provides%20two%20primary%20examples%2C%20one%20using%20custom%20machine%20learning%20and%20the%20other%20using%20AutoML.%20%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fsolution-accelerator-many-models%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EGive%20it%20a%20try%20today!%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1189474572%22%20id%3D%22toc-hId-1189474572%22%20id%3D%22toc-hId-1189474572%22%3E%3CEM%3EExecutive%20Overview%3C%2FEM%3E%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EMany%20executives%20are%20looking%20to%20Machine%20Learning%20to%20improve%20their%20business.%20With%20the%20reliance%20on%20a%20more%20digital%20world%2C%20the%20amount%20of%20data%20generated%20is%20increasing%20faster%20than%20ever.%20Further%2C%20companies%20are%20purchasing%203%3CSUP%3Erd%3C%2FSUP%3E%20party%20datasets%20to%20combine%20with%20internal%20data%20to%20gain%20further%20insight%20and%20make%20better%20predictions.%20In%20order%20to%20make%20better%20predictions%2C%20sophisticated%20machine%20learning%20models%20are%20being%20built%20that%20leverages%20this%20large%20pool%20of%20data.%20Further%2C%20as%20companies%20are%20expanding%20to%20do%20business%20in%20a%20variety%20of%20markets%20and%20environments%2C%20general%20machine%20learning%20model%20no%20longer%20suffice%2C%20instead%2C%20more%20specific%20machine%20learning%20models%20are%20needed.%20In%20this%20case%2C%20%22general%20machine%20learning%20models%22%20refers%20to%20the%20granularity%20for%20which%20this%20model%20was%20built%2C%20for%20example%2C%20building%20a%20demand%20forecast%20model%20for%20a%20product%20at%20the%20country%20level%20versus%20at%20the%20city%20level%20which%20would%20be%20a%20%22specific%20machine%20learning%20model.%22%26nbsp%3B%20Building%20more%20specific%20machine%20learning%20models%20can%20easily%20results%20in%20building%20hundreds%20of%20thousands%20of%20specific%20models%20instead%20of%20a%20handful%20of%20general%20models.%20%26nbsp%3BCombining%20large%20datasets%20with%20the%20need%20of%20building%20hundreds%20of%20thousands%20of%20more%20specific%20machine%20learning%20models%20is%20not%20a%20trivial%20task.%20Doing%20so%20requires%20very%20large%20compute%20power.%20In%20fact%2C%20this%20task%20can%20greatly%20benefit%20from%20parallel%20compute%20power%20where%20multiple%20compute%20instances%20are%20working%20simultaneously%20to%20build%20the%20machine%20learning%20models%20in%20parallel.%20Once%20those%20models%20are%20trained%2C%20leveraging%20them%20to%20score%20large%20amounts%20of%20data%20presents%20the%20same%20problem%20characteristics%20which%20again%2C%20leveraging%20a%20compute%20cluster%20where%20multiple%20compute%20instances%20are%20making%20predictions%20using%20the%20machine%20learning%20models%20simultaneously%20can%20greatly%20reduce%20the%20time%20required%20to%20do%20so.%20With%20the%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2F%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAzure%20Machine%20Learning%20service%3C%2FA%3E%2C%20the%20training%20and%20scoring%20of%20hundreds%20of%20thousands%20of%20models%20with%20large%20amounts%20of%20data%20can%20be%20completed%20efficiently%20leveraging%20pipelines%20where%20certain%20steps%20like%20model%20training%20and%20model%20scoring%20run%20in%20parallel%20on%20large%20scale%20out%20compute%20clusters.%20In%20order%20to%20help%20organizations%20get%20a%20head%20start%20on%20building%20such%20pipelines%2C%20the%20%5B%23%24dp7%5D%3CEM%3E%5B%23%24dp19%5D%3CEM%3EMany%20Models%20Solution%20Accelerator%3C%2FEM%3E%3C%2FEM%3E%20has%20been%20created.%20The%20Many%20Models%20Solution%20Accelerator%20provides%20two%20primary%20examples%2C%20one%20using%20custom%20machine%20learning%20and%20the%20other%20using%20AutoML.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20Azure%20Machine%20Learning%2C%20AutoML%20automates%20the%20building%20of%20the%20most%20common%20categories%20of%20Machine%20Learning%20models%20in%20a%20very%20robust%20and%20sophisticated%20manner.%20For%20example%2C%20a%20very%20common%20machine%20learning%20problem%20is%20demand%20forecasting.%20Having%20a%20more%20accurate%20demand%20forecast%20can%20increase%20revenues%20and%20reduce%20waste.%20Traditionally%2C%20many%20statistical%20methods%20have%20been%20used%20to%20do%20just%20this.%20However%2C%20more%20modern%20techniques%20leverage%20Machine%20Learning%20including%20Deep%20Learning%20techniques%20to%20provide%20a%20more%20accurate%20demand%20forecast.%20Further%2C%20the%20demand%20forecast%20can%20be%20improved%20by%20moving%20from%20forecasting%20a%20broader%20scope%20(general%20machine%20learning%20model)%20to%20forecasting%20a%20more%20granular%20scope%20(specific%20machine%20learning%20model).%26nbsp%3B%20Doing%20so%20means%2C%20for%20example%2C%20instead%20of%20building%20one%20forecast%20for%20each%20Product%20at%20the%20country%20level%2C%20building%20a%20forecast%20for%20each%20product%20at%20the%20city.%20Moving%20to%20more%20specific%20models%20results%20in%20building%20hundreds%20of%20thousands%20of%20forecasts%20using%20large%20amounts%20of%20data%20which%20as%20discussed%20above%20can%20be%20solved%20using%20the%20%3CEM%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fsolution-accelerator-many-models%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMany%20Models%20Solution%20Accelerator%3C%2FA%3E.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--617979891%22%20id%3D%22toc-hId--617979891%22%20id%3D%22toc-hId--617979891%22%3E%3CEM%3ETechnical%20Overview%3C%2FEM%3E%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAs%20data%20scientist%20move%20from%20building%20a%20handful%20of%20general%20machine%20learning%20models%20to%20hundreds%20of%20thousands%20of%20more%20specific%20machine%20learning%20models%20(i.e.%20geography%20or%20product%20scope)%2C%20the%20need%20to%20perform%20the%20model%20training%20and%20model%20scoring%20tasks%20require%20parallel%20compute%20power%20to%20finish%20in%20a%20timely%20manner.%20In%20the%20Azure%20Machine%20Learning%20service%20SDK%2C%20this%20is%20accomplished%20using%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpython%2Fapi%2Fazureml-pipeline-core%2Fazureml.pipeline.core.pipeline.pipeline%3Fview%3Dazure-ml-py%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EPipelines%3C%2FA%3E%20and%20specifically%20a%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpython%2Fapi%2Fazureml-contrib-pipeline-steps%2Fazureml.contrib.pipeline.steps.parallel_run_step.parallelrunstep%3Fview%3Dazure-ml-py%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EParallelRunStep%3C%2FA%3E%20which%20runs%20on%20a%20multi%20node%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2Fhow-to-set-up-training-targets%23amlcompute%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ECompute%20Clusters%3C%2FA%3E.%20The%20data%20scientist%20provides%20the%20ParallelRunStep%20a%20custom%20script%2C%20an%20input%20dataset%2C%20a%20compute%20cluster%20and%20the%20amount%20of%20parallelism%20they%20would%20like.%20This%20concept%20can%20be%20applied%20to%20a%20custom%20python%20script%20and%20to%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2Fhow-to-configure-auto-train%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAutomated%20Machine%20Learning%20(AutoML)%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2Fhow-to-configure-auto-train%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAutomated%20Machine%20Learning%20(AutoML)%3C%2FA%3E%20uses%20over%20ten%20algorithms%20(including%20deep%20learning%20algorithms)%20with%20varying%20hyperparameters%20to%20build%20Classification%2C%20Regression%20and%20Forecasting%20models.%20Further%2C%20-ERR%3AREF-NOT-FOUND-Pipelines%20automate%20the%20invocation%20of%20AutoML%20across%20multiple%20nodes%20using%26nbsp%3B%20-ERR%3AREF-NOT-FOUND-ParallelRunStep%20to%20train%20the%20models%20in%20parallel%20as%20well%20as%20to%20-ERR%3AREF-NOT-FOUND-batch%20score%20new%20data.%20Pipelines%20can%20be%20scheduled%20to%20run%20within%20Azure%20Machine%20Learning%20or%20invoked%20using%20their%20REST%20endpoint%20from%20various%20Azure%20services%20(i.e.%20Azure%20Data%20Factory%2C%20Azure%20DevOps%2C%20Azure%20Functions%2C%20Azure%20Logic%20Apps%2C%20etc).%20When%20invoked%2C%20the%20Parallel%20Pipelines%20run%20on%20-ERR%3AREF-NOT-FOUND-Compute%20Clusters%20within%20Azure%20Machine%20Learning.%20The%20Compute%20clusters%20can%20be%20scaled%20up%20and%20out%20to%20perform%20the%20training%20and%20scoring.%20Each%20node%20in%20a%20compute%20clusters%20can%20be%20have%20Terabytes%20of%20RAM%2C%20over%20a%20100%20cores%2C%20and%20multiple%20GPUs.%20Finally%2C%20the%20scored%20data%20can%20be%20stored%20in%20an%20a%20datastore%20in%20Azure%2C%20such%20as%20Azure%20DataLake%20Gen%202%2C%20and%20then%20copied%20to%20a%20specific%20location%20for%20an%20application%20to%20consume%20the%20results.%3C%2FP%3E%0A%3CP%3EIn%20order%20to%20provide%20a%20jump%20start%20in%20leveraging%20Pipelines%20with%20the%20new%20ParallelRunStep%2C%20the%20-ERR%3AREF-NOT-FOUND-%3CEM%3EMany%20Models%20Solution%20Accelerator%3C%2FEM%3E%20has%20been%20created.%20This%20solution%20accelerator%20showcases%20both%20a%20custom%20python%20script%20and%20an%20AutoML%20script.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1869532942%22%20id%3D%22toc-hId-1869532942%22%20id%3D%22toc-hId-1869532942%22%3E%3CEM%3EMajor%20Components%3C%2FEM%3E%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20main%20components%20of%20the%20Many%20Models%20Solution%20Accelerator%20includes%20an%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2Fconcept-workspace%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAzure%20Machine%20Learning%20Workspace%3C%2FA%3E%2C%20a%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2Fconcept-ml-pipelines%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EPipeline%3C%2FA%3E%2C%20a%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpython%2Fapi%2Fazureml-contrib-pipeline-steps%2Fazureml.contrib.pipeline.steps.parallel_run_step.parallelrunstep%3Fview%3Dazure-ml-py%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EParallelRunStep%3C%2FA%3E%2C%20a%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2Fconcept-compute-target%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ECompute%20Target%3C%2FA%3E%2C%20a%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2Fconcept-data%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EDatastore%3C%2FA%3E%2C%20and%20a%20Python%20Script%20File%20as%20depicted%20in%20Figure%201%2C%20below.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22Sam_Istephan_0-1595858410872.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F208186i774F39D8DF91F0E4%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22Sam_Istephan_0-1595858410872.png%22%20alt%3D%22Sam_Istephan_0-1595858410872.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-align-left%22%3EFigure%201.%20The%20architecture%20of%20a%20Pipeline%20with%20a%20ParallelRunStep%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20an%20overview%20of%20getting%20started%20with%20Azure%20Machine%20Learning%20service%2C%20please%20see%20the%20blog%20article%20%3CA%20href%3D%22https%3A%2F%2Fmlonazure.com%2Fgettingstartedwithaml%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMLonAzure%3A%20Getting%20Started%20with%20Azure%20Machine%20Learning%20service%20for%20the%20Data%20Scientist%3C%2FA%3E%3C%2FP%3E%0A%3CP%3EFor%20an%20overview%20of%20Pipelines%2C%20please%20see%20blog%20article%2C%20-ERR%3AREF-NOT-FOUND-MLonAzure%3A%20Azure%20Machine%20Learning%20service%20Pipelines%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-62078479%22%20id%3D%22toc-hId-62078479%22%20id%3D%22toc-hId-62078479%22%3E%3CEM%3EMajor%20Steps%3C%2FEM%3E%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH3%20id%3D%22toc-hId-752639953%22%20id%3D%22toc-hId-752639953%22%20id%3D%22toc-hId-752639953%22%3E1.%20Prerequisites%3C%2FH3%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EAn%20Azure%20Subscription%20along%20with%20an%20Azure%20Machine%20Learning%20Workspace%20are%20needed%20to%20get%20started.%20To%20get%20started%20with%20Azure%20Machine%20Learning%2C%20see%20-ERR%3AREF-NOT-FOUND-Getting%20Started%20with%20Azure%20Machine%20Learning%20services%3C%2FLI%3E%0A%3CLI%3EWithin%20the%20Azure%20Machine%20Learning%2C%20a%20Compute%20Instance%20needs%20to%20be%20created%20to%20serve%20as%20the%20Data%20Scientist%E2%80%99s%20workstation.%20Using%20the%20Compute%20Instance%2C%20clone%20the%20%3CA%20href%3D%22https%3A%2F%2Fmlonazure.com%2Fpipelines%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMany%20Models%20Solution%20Accelerator%3C%2FA%3E%20Github%20repository.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CH2%20id%3D%22toc-hId-742136849%22%20id%3D%22toc-hId-742136849%22%20id%3D%22toc-hId-742136849%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH3%20id%3D%22toc-hId-1432698323%22%20id%3D%22toc-hId-1432698323%22%20id%3D%22toc-hId-1432698323%22%3E2.%26nbsp%3B%20Data%20Prep%3C%2FH3%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EData%20needs%20to%20be%20split%20into%20multiple%20files%20(.csv%20or%20.parquet)%20for%20each%20group%20that%20a%20model%20is%20to%20be%20created%20for.%20Each%20file%20must%20contain%20one%20or%20more%20entire%20time%20series%20for%20the%20give%20group.%3C%2FLI%3E%0A%3CLI%3EThe%20data%20must%20be%20placed%20in%20Azure%20Storage%20(e.g.%20ADL%20Gen%202%2C%20Blob%20Storage).%20The%20storage%20will%20then%20be%20registered%20as%20a%20-ERR%3AREF-NOT-FOUND-Datastore%20from%20which%20two%20-ERR%3AREF-NOT-FOUND-FileDatasets%20will%20be%20registered%2C%20one%20pointing%20to%20the%20folder%20containing%20the%20training%20data%20and%20the%20other%20to%20the%20folder%20containing%20the%20data%20to%20be%20scored.%3C%2FLI%3E%0A%3CLI%3EFor%20example%2C%20to%20build%20a%20forecast%20model%20for%20each%20brand%20within%20a%20store%2C%20the%20training%20sales%20data%20would%20be%20split%20to%20create%20files%20StoreXXXX_BrandXXXX.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CH2%20id%3D%22toc-hId-1422195219%22%20id%3D%22toc-hId-1422195219%22%20id%3D%22toc-hId-1422195219%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH3%20id%3D%22toc-hId-2112756693%22%20id%3D%22toc-hId-2112756693%22%20id%3D%22toc-hId-2112756693%22%3E3.%20Model%20Training%3C%2FH3%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20solution%20accelerator%20showcases%20model%20training%20with%20a%20custom%20python%20script%20and%20with%20AutoML%20which%20are%20orchestrated%20using%20a%20-ERR%3AREF-NOT-FOUND-Pipeline.%20Please%20see%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpython%2Fapi%2Fazureml-core%2Fazureml.core.datastore.datastore%3Fview%3Dazure-ml-py%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Esolution-accelerator-manymodels-customscript%3C%2FA%3E%20and%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpython%2Fapi%2Fazureml-core%2Fazureml.data.filedataset%3Fview%3Dazure-ml-py%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Esolution-accelerator-manymodels-AutoML%3C%2FA%3E.%20Putting%20it%20all%20together%20results%20in%20the%20architecture%20depicted%20in%20Figure%202%2C%20below.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22Sam_Istephan_2-1595858410885.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F208187iB9C9AEC4D7A878C6%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22Sam_Istephan_2-1595858410885.png%22%20alt%3D%22Sam_Istephan_2-1595858410885.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-align-left%22%3EFigure%202%3A%20Solution%20Accelerator%20Model%20Training%3C%2FP%3E%0A%3CP%20class%3D%22lia-align-left%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CH4%20id%3D%22toc-hId--260778530%22%20id%3D%22toc-hId--260778530%22%20id%3D%22toc-hId--260778530%22%3E3.%20a.%20Pipeline%3C%2FH4%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20solution%20accelerator%20leverages%20the%20%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fsolution-accelerator-many-models%2Ftree%2Fmaster%2FCustom_Script%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EPipeline%3C%2FA%3E%20object%20to%20train%20the%20model.%20Specifically%2C%20a%20%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fsolution-accelerator-many-models%2Ftree%2Fmaster%2FAutomated_ML%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EParallelRunStep%3C%2FA%3E%20is%20used%20which%20requires%20a%20configuration%20parameters%2C%20-ERR%3AREF-NOT-FOUND-ParallelRunConfig%3C%2FP%3E%0A%3CP%3E-ERR%3AREF-NOT-FOUND-ParallelRunConfig%20has%20many%20parameters%2C%20below%20are%20the%20typical%20ones%20used%20for%20the%20Many%20Models%20Solution%20Accelerator.%20For%20a%20complete%20list%20of%20-ERR%3AREF-NOT-FOUND-ParallelRunConfig%20parameters%2C%20please%20see%20the%20-ERR%3AREF-NOT-FOUND-%26nbsp%3BParallelRunConfig%20Class.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CTABLE%3E%0A%3CTBODY%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216%22%3E%3CP%3EParameter%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408%22%3E%3CP%3EExplanation%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216%22%3E%3CP%3Eenvironment%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408%22%3E%3CP%3EProvides%20the%20configurations%20for%20the%20Python%20Environment%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216%22%3E%3CP%3EEntry_script%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408%22%3E%3CP%3EThis%20is%20a%20Python%20file%20(.py%20extension%20only)%20that%20will%20run%20in%20parallel.%20Note%20that%20the%20Many%20Models%20Solution%20Accelerator%20contains%20a%20custom%20Entry_script%20that%20leverages%20AutoML%20and%20one%20that%20leverages%20custom%20code.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216%22%3E%3CP%3ECompute_target%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408%22%3E%3CP%3EThe%20AML%20ComputeCluster%20to%20run%20the%20step%20on.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216%22%3E%3CP%3ENode_count%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408%22%3E%3CP%3EThe%20number%20of%20nodes%20to%20use%20within%20the%20training%20cluster.%20Scale%20this%20number%20to%20a%20higher%20number%20to%20increase%20parallelism.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216%22%3E%3CP%3EProcess_count_per_node%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408%22%3E%3CP%3EThe%20number%20of%20cores%20that%20will%20be%20used%20within%20each%20node%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216%22%3E%3CP%3Emini_batch_size%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408%22%3E%3CP%3EFor%20FileDatasets%20it%E2%80%99s%20the%20number%20of%20files%20that%20are%20used%20at%20a%20time%2C%20for%20Tabular%20Datasets%20it%E2%80%99s%20the%20number%20data%20size%20that%20will%20be%20processed%20at%20a%20given%20time.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216%22%3E%3CP%3ERun_inovcation_timeout%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408%22%3E%3CP%3EThe%20overall%20allowed%20time%20for%20the%20parallel%20run%20step%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3C%2FTBODY%3E%0A%3C%2FTABLE%3E%0A%3CH3%20id%3D%22toc-hId-2097651584%22%20id%3D%22toc-hId-2097651584%22%20id%3D%22toc-hId-2097651584%22%3E%26nbsp%3B%3C%2FH3%3E%0A%3CH4%20id%3D%22toc-hId-419279840%22%20id%3D%22toc-hId-419279840%22%20id%3D%22toc-hId-419279840%22%3E3.%20b.%20Training%20script%20with%20AutoML%3C%2FH4%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20solution%20accelerator%20showcases%20using%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2Fhow-to-auto-train-forecast%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAutoML%20Forecasting%3C%2FA%3E.%20AutoML%20has%20many%20parameters%2C%20below%20are%20the%20typical%20ones%20used%20for%20doing%20a%20Forecasting%20task%20within%20the%20Many%20Models%20Solution%20Accelerator.%20For%20a%20complete%20list%20of%20AutoML%20Config%20parameters%2C%20please%20see%20the%20%5B%23%24dp206%5D%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpython%2Fapi%2Fazureml-train-automl-client%2Fazureml.train.automl.automlconfig.automlconfig%3Fview%3Dazure-ml-py%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAutoMLConfig%20Class%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-applescript%22%3E%3CCODE%3Eautoml_settings%20%3D%20%7B%0A%20%20%20%20%22task%22%20%3A%20'forecasting'%2C%0A%20%20%20%20%22primary_metric%22%20%3A%20'normalized_root_mean_squared_error'%2C%0A%20%20%20%20%22iteration_timeout_minutes%22%20%3A%2010%2C%20%0A%20%20%20%20%22iterations%22%20%3A%2015%2C%0A%20%20%20%20%22experiment_timeout_hours%22%20%3A%201%2C%0A%20%20%20%20%22label_column_name%22%20%3A%20'Quantity'%2C%0A%20%20%20%20%22n_cross_validations%22%20%3A%203%2C%0A%20%20%20%20%22verbosity%22%20%3A%20logging.INFO%2C%20%0A%20%20%20%20%22debug_log%22%3A%20'DebugFileName.txt'%2C%0A%20%20%20%20%22time_column_name%22%3A%20'WeekStarting'%2C%0A%20%20%20%20%22max_horizon%22%20%3A%206%2C%0A%20%20%20%20%22group_column_names%22%3A%20%5B'Store'%2C%20'Brand'%5D%2C%0A%20%20%20%20%22grain_column_names%22%3A%20%5B'Store'%2C%20'Brand'%5D%0A%7D%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CTABLE%3E%0A%3CTBODY%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2230px%22%3E%0A%3CP%3EParameter%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2230px%22%3E%0A%3CP%3EExplanation%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2257px%22%3E%0A%3CP%3Etask%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2257px%22%3E%0A%3CP%3EThe%20type%20of%20AutoML%20Task%3A%20Classification%2C%20Regression%2C%20Forecasting%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2230px%22%3E%0A%3CP%3Eprimary_metric%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2230px%22%3E%0A%3CP%3EThe%20metric%20that%20AutoML%20should%20optimize%20based%20on%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2230px%22%3E%0A%3CP%3EIteration_timeout_minutes%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2230px%22%3E%0A%3CP%3EHow%20long%20each%20of%20the%20number%20of%20%E2%80%9Citerations%E2%80%9D%20can%20run%20for%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2257px%22%3E%0A%3CP%3EIterations%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2257px%22%3E%0A%3CP%3ENumber%20of%20models%20that%20should%20be%20tried%20(combinations%20of%20various%20Algorithms%20%2B%20various%20Hyperparameters)%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2284px%22%3E%0A%3CP%3EExperiment_timeout_hours%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2284px%22%3E%0A%3CP%3EHow%20long%20the%20overall%20AutoML%20Experiment%20can%20take.%20Note%3A%20The%20experiment%20may%20might%20timeout%20before%20all%20iterations%20are%20complete.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2230px%22%3E%0A%3CP%3Elabel_column_name%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2230px%22%3E%0A%3CP%3EThe%20column%20that%20is%20being%20predicted%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2257px%22%3E%0A%3CP%3En_cross_validations%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2257px%22%3E%0A%3CP%3ENumber%20of%20cross%20validations%20that%20should%20take%20place%20within%20the%20training%20dataset%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2230px%22%3E%0A%3CP%3Everbosity%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2230px%22%3E%0A%3CP%3ELog%20details%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2230px%22%3E%0A%3CP%3Edebug_log%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2230px%22%3E%0A%3CP%3ELocation%20for%20the%20debug%20log%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2257px%22%3E%0A%3CP%3Etime_column_name%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2257px%22%3E%0A%3CP%3EThe%20name%20of%20the%20Time%20column%2C%20note%20that%20the%20training%20dataset%20can%20have%20multiple%20time%20series%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2230px%22%3E%0A%3CP%3Emax_horizon%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2230px%22%3E%0A%3CP%3EHow%20far%20how%20the%20forecast%20will%20go%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2257px%22%3E%0A%3CP%3Egroup_column_names%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2257px%22%3E%0A%3CP%3EThe%20names%20of%20columns%20used%20to%20group%20your%20models.%20For%20timeseries%2C%20the%20groups%20must%20not%20split%20up%20individual%20time-series.%20That%20is%2C%20each%20group%20must%20contain%20one%20or%20more%20whole%20time-series.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22216px%22%20height%3D%2230px%22%3E%0A%3CP%3Egrain_column_names%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22408px%22%20height%3D%2230px%22%3E%0A%3CP%3EThe%20column%20names%20used%20to%20uniquely%20identify%20timeseries%20in%20data%20that%20has%20multiple%20rows%20with%20the%20same%20timestamp.%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3C%2FTBODY%3E%0A%3C%2FTABLE%3E%0A%3CH2%20id%3D%22toc-hId--1646340061%22%20id%3D%22toc-hId--1646340061%22%20id%3D%22toc-hId--1646340061%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH3%20id%3D%22toc-hId-970255491%22%20id%3D%22toc-hId-970255491%22%20id%3D%22toc-hId-970255491%22%3E4.%20Model%20Forecasting%3C%2FH3%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20solution%20accelerator%20showcases%20model%20forecasting%20with%20a%20custom%20python%20script%20and%20with%20AutoML%20which%20are%20orchestrated%20using%20a%20-ERR%3AREF-NOT-FOUND-Pipeline.%20Please%20see%20-ERR%3AREF-NOT-FOUND-solution-accelerator-manymodels-customscript%20and%20-ERR%3AREF-NOT-FOUND-solution-accelerator-manymodels-AutoML.%20Putting%20it%20all%20together%20results%20in%20the%20architecture%20depicted%20in%20Figure%203%2C%20below.%3C%2FP%3E%0A%3CDIV%20id%3D%22tinyMceEditorSam_Istephan_6%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorSam_Istephan_0%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22Sam_Istephan_4-1595858410900.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F208189i5EEB941EF199227F%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22Sam_Istephan_4-1595858410900.png%22%20alt%3D%22Sam_Istephan_4-1595858410900.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-align-left%22%3EFigure%203%3A%20Solution%20Accelerator%20Model%20Scoring%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--966281691%22%20id%3D%22toc-hId--966281691%22%20id%3D%22toc-hId--966281691%22%3E%26nbsp%3B%3C%2FH2%3E%0A%3CH3%20id%3D%22toc-hId-1650313861%22%20id%3D%22toc-hId-1650313861%22%20id%3D%22toc-hId-1650313861%22%3E5.%20Automation%3C%2FH3%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20order%20to%20automate%20the%20solution%2C%20the%20training%20and%20scoring%20pipelines%20must%20be%20published%20and%20a%20PipelineEndPoint%20must%20be%20created.%20Once%20that%E2%80%99s%20done%2C%20the%20-ERR%3AREF-NOT-FOUND-PipelineEndpoint%20can%20then%20be%20invoked%20from%20-ERR%3AREF-NOT-FOUND-Azure%20Data%20Factory.%20Specifically%2C%20the%20-ERR%3AREF-NOT-FOUND-Azure%20Machine%20Learning%20Pipeline%20Activity%20is%20used.%20Note%20that%20the%20training%20and%20scoring%20pipelines%20can%20be%20collapsed%20into%20one%20pipelines%20if%20the%20training%20and%20scoring%20occur%20consecutively.%20%26nbsp%3B%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId--415306040%22%20id%3D%22toc-hId--415306040%22%20id%3D%22toc-hId--415306040%22%3E%26nbsp%3B%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--2093677784%22%20id%3D%22toc-hId--2093677784%22%20id%3D%22toc-hId--2093677784%22%3E%3CEM%3ENext%20Steps%3C%2FEM%3E%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2F%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAzure%20Machine%20Learning%20Documentation%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fsolution-accelerator-many-models%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMany%20Models%20Solution%20Accelerator%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fchannel9.msdn.com%2FShows%2FDocs-AI%2FBuilding-Large-Scale-Machine-Learning-Models-using-Azure-Machine-Learning%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMany%20Models%20Solution%20Accelerator%20Video%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Ftransform-data-machine-learning-service%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAzure%20Data%20Factory%3A%20Azure%20Machine%20Learning%20Pipeline%20Activity%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fmlonazure%2FAzureMachineLearning%2Ftree%2Fmaster%2FPipeline%2520Python%2520and%2520R%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMLOnAzure%20GitHub%3A%20Getting%20started%20with%20Pipelines%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fmlonazure.com%2Fgettingstartedwithaml%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMLonAzure%20Blog%3A%20Getting%20Started%20with%20Azure%20Machine%20Learning%20for%20the%20Data%20Scientist%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fmlonazure.com%2Fpipelines%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMLonAzure%20Blog%3A%20Azure%20Machine%20Learning%20service%20Pipelines%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fmlonazure.com%2Fpipelines-parallelrunstep%2F%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMLonAzureBlog%3A%20Azure%20Machine%20Learning%20Pipeline%20-%20ParallelRunStep%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fyoutu.be%2FwnB-EpjALIQ%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMLonAzureBlog%3A%20Video%20Walkthrough%20of%20ParallelRunStep%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fwww.linkedin.com%2Fgroups%2F12405375%2F%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ELinkedIn%20Group%3A%20Machine%20Learning%20on%20Azure%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1547960%22%20slang%3D%22en-US%22%3E%3CP%3EWith%20the%20Azure%20Machine%20Learning%20service%2C%20the%20training%20and%20scoring%20of%20hundreds%20of%20thousands%20of%20models%20with%20large%20amounts%20of%20data%20can%20be%20completed%20efficiently%20leveraging%20pipelines%20where%20certain%20steps%20like%20model%20training%20and%20model%20scoring%20run%20in%20parallel%20on%20large%20scale%20out%20compute%20clusters.%20In%20order%20to%20help%20organizations%20get%20a%20head%20start%20on%20building%20such%20pipelines%2C%20the%20%5B%23%24dp7%5D%3CEM%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fsolution-accelerator-many-models%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%20target%3D%22_blank%22%3E%3CEM%3EMany%20Models%20Solution%20Accelerator%3C%2FEM%3E%3C%2FA%3E%3C%2FEM%3E%20has%20been%20created.%20The%20Many%20Models%20Solution%20Accelerator%20provides%20two%20primary%20examples%2C%20one%20using%20custom%20machine%20learning%20and%20the%20other%20using%20AutoML.%20%5B%23%24dp415%5DGive%20it%20a%20try%20today!%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1547960%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Machine%20Learning%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMachine%20Learning%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMLOps%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1572661%22%20slang%3D%22en-US%22%3ERe%3A%20Train%20and%20Score%20Hundreds%20of%20Thousands%20of%20Models%20in%20Parallel%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1572661%22%20slang%3D%22en-US%22%3E%3CP%3EThanks%20for%20sharing!%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

Abstract

 

With the Azure Machine Learning service, the training and scoring of hundreds of thousands of models with large amounts of data can be completed efficiently leveraging pipelines where certain steps like model training and model scoring run in parallel on large scale out compute clusters. In order to help organizations get a head start on building such pipelines, the Many Models Solution Accelerator has been created. The Many Models Solution Accelerator provides two primary examples, one using custom machine learning and the other using AutoML. Give it a try today!

 

Executive Overview

 

Many executives are looking to Machine Learning to improve their business. With the reliance on a more digital world, the amount of data generated is increasing faster than ever. Further, companies are purchasing 3rd party datasets to combine with internal data to gain further insight and make better predictions. In order to make better predictions, sophisticated machine learning models are being built that leverages this large pool of data. Further, as companies are expanding to do business in a variety of markets and environments, general machine learning model no longer suffice, instead, more specific machine learning models are needed. In this case, "general machine learning models" refers to the granularity for which this model was built, for example, building a demand forecast model for a product at the country level versus at the city level which would be a "specific machine learning model."  Building more specific machine learning models can easily results in building hundreds of thousands of specific models instead of a handful of general models.  Combining large datasets with the need of building hundreds of thousands of more specific machine learning models is not a trivial task. Doing so requires very large compute power. In fact, this task can greatly benefit from parallel compute power where multiple compute instances are working simultaneously to build the machine learning models in parallel. Once those models are trained, leveraging them to score large amounts of data presents the same problem characteristics which again, leveraging a compute cluster where multiple compute instances are making predictions using the machine learning models simultaneously can greatly reduce the time required to do so. With the Azure Machine Learning service, the training and scoring of hundreds of thousands of models with large amounts of data can be completed efficiently leveraging pipelines where certain steps like model training and model scoring run in parallel on large scale out compute clusters. In order to help organizations get a head start on building such pipelines, the Many Models Solution Accelerator has been created. The Many Models Solution Accelerator provides two primary examples, one using custom machine learning and the other using AutoML.

 

In Azure Machine Learning, AutoML automates the building of the most common categories of Machine Learning models in a very robust and sophisticated manner. For example, a very common machine learning problem is demand forecasting. Having a more accurate demand forecast can increase revenues and reduce waste. Traditionally, many statistical methods have been used to do just this. However, more modern techniques leverage Machine Learning including Deep Learning techniques to provide a more accurate demand forecast. Further, the demand forecast can be improved by moving from forecasting a broader scope (general machine learning model) to forecasting a more granular scope (specific machine learning model).  Doing so means, for example, instead of building one forecast for each Product at the country level, building a forecast for each product at the city. Moving to more specific models results in building hundreds of thousands of forecasts using large amounts of data which as discussed above can be solved using the Many Models Solution Accelerator.

 

Technical Overview

 

As data scientist move from building a handful of general machine learning models to hundreds of thousands of more specific machine learning models (i.e. geography or product scope), the need to perform the model training and model scoring tasks require parallel compute power to finish in a timely manner. In the Azure Machine Learning service SDK, this is accomplished using Pipelines and specifically a ParallelRunStep which runs on a multi node Compute Clusters. The data scientist provides the ParallelRunStep a custom script, an input dataset, a compute cluster and the amount of parallelism they would like. This concept can be applied to a custom python script and to Automated Machine Learning (AutoML).

 

Automated Machine Learning (AutoML) uses over ten algorithms (including deep learning algorithms) with varying hyperparameters to build Classification, Regression and Forecasting models. Further, Pipelines automate the invocation of AutoML across multiple nodes using  ParallelRunStep to train the models in parallel as well as to batch score new data. Pipelines can be scheduled to run within Azure Machine Learning or invoked using their REST endpoint from various Azure services (i.e. Azure Data Factory, Azure DevOps, Azure Functions, Azure Logic Apps, etc). When invoked, the Parallel Pipelines run on Compute Clusters within Azure Machine Learning. The Compute clusters can be scaled up and out to perform the training and scoring. Each node in a compute clusters can be have Terabytes of RAM, over a 100 cores, and multiple GPUs. Finally, the scored data can be stored in an a datastore in Azure, such as Azure DataLake Gen 2, and then copied to a specific location for an application to consume the results.

In order to provide a jump start in leveraging Pipelines with the new ParallelRunStep, the Many Models Solution Accelerator has been created. This solution accelerator showcases both a custom python script and an AutoML script.

 

Major Components

 

The main components of the Many Models Solution Accelerator includes an Azure Machine Learning Workspace, a Pipeline, a ParallelRunStep, a Compute Target, a Datastore, and a Python Script File as depicted in Figure 1, below.

 

Sam_Istephan_0-1595858410872.png

Figure 1. The architecture of a Pipeline with a ParallelRunStep

 

For an overview of getting started with Azure Machine Learning service, please see the blog article MLonAzure: Getting Started with Azure Machine Learning service for the Data Scientist

For an overview of Pipelines, please see blog article, MLonAzure: Azure Machine Learning service Pipelines

 

Major Steps

 

1. Prerequisites

 

 

2.  Data Prep

 

  • Data needs to be split into multiple files (.csv or .parquet) for each group that a model is to be created for. Each file must contain one or more entire time series for the give group.
  • The data must be placed in Azure Storage (e.g. ADL Gen 2, Blob Storage). The storage will then be registered as a Datastore from which two FileDatasets will be registered, one pointing to the folder containing the training data and the other to the folder containing the data to be scored.
  • For example, to build a forecast model for each brand within a store, the training sales data would be split to create files StoreXXXX_BrandXXXX.

 

3. Model Training

 

The solution accelerator showcases model training with a custom python script and with AutoML which are orchestrated using a Pipeline. Please see solution-accelerator-manymodels-customscript and solution-accelerator-manymodels-AutoML. Putting it all together results in the architecture depicted in Figure 2, below.

 

Sam_Istephan_2-1595858410885.png

Figure 2: Solution Accelerator Model Training

 

3. a. Pipeline

 

The solution accelerator leverages the Pipeline object to train the model. Specifically, a ParallelRunStep is used which requires a configuration parameters, ParallelRunConfig

ParallelRunConfig has many parameters, below are the typical ones used for the Many Models Solution Accelerator. For a complete list of ParallelRunConfig parameters, please see the  ParallelRunConfig Class.

 

Parameter

Explanation

environment

Provides the configurations for the Python Environment

Entry_script

This is a Python file (.py extension only) that will run in parallel. Note that the Many Models Solution Accelerator contains a custom Entry_script that leverages AutoML and one that leverages custom code.

Compute_target

The AML ComputeCluster to run the step on.

Node_count

The number of nodes to use within the training cluster. Scale this number to a higher number to increase parallelism.

Process_count_per_node

The number of cores that will be used within each node

mini_batch_size

For FileDatasets it’s the number of files that are used at a time, for Tabular Datasets it’s the number data size that will be processed at a given time.

Run_inovcation_timeout

The overall allowed time for the parallel run step

 

3. b. Training script with AutoML

 

The solution accelerator showcases using AutoML Forecasting. AutoML has many parameters, below are the typical ones used for doing a Forecasting task within the Many Models Solution Accelerator. For a complete list of AutoML Config parameters, please see the AutoMLConfig Class.

 

 

 

automl_settings = {
    "task" : 'forecasting',
    "primary_metric" : 'normalized_root_mean_squared_error',
    "iteration_timeout_minutes" : 10, 
    "iterations" : 15,
    "experiment_timeout_hours" : 1,
    "label_column_name" : 'Quantity',
    "n_cross_validations" : 3,
    "verbosity" : logging.INFO, 
    "debug_log": 'DebugFileName.txt',
    "time_column_name": 'WeekStarting',
    "max_horizon" : 6,
    "group_column_names": ['Store', 'Brand'],
    "grain_column_names": ['Store', 'Brand']
}

 

 

 

Parameter

Explanation

task

The type of AutoML Task: Classification, Regression, Forecasting

primary_metric

The metric that AutoML should optimize based on

Iteration_timeout_minutes

How long each of the number of “iterations” can run for

Iterations

Number of models that should be tried (combinations of various Algorithms + various Hyperparameters)

Experiment_timeout_hours

How long the overall AutoML Experiment can take. Note: The experiment may might timeout before all iterations are complete.

label_column_name

The column that is being predicted

n_cross_validations

Number of cross validations that should take place within the training dataset

verbosity

Log details

debug_log

Location for the debug log

time_column_name

The name of the Time column, note that the training dataset can have multiple time series

max_horizon

How far how the forecast will go

group_column_names

The names of columns used to group your models. For timeseries, the groups must not split up individual time-series. That is, each group must contain one or more whole time-series.

grain_column_names

The column names used to uniquely identify timeseries in data that has multiple rows with the same timestamp.

 

4. Model Forecasting

 

The solution accelerator showcases model forecasting with a custom python script and with AutoML which are orchestrated using a Pipeline. Please see solution-accelerator-manymodels-customscript and solution-accelerator-manymodels-AutoML. Putting it all together results in the architecture depicted in Figure 3, below.

 
 

Sam_Istephan_4-1595858410900.png

Figure 3: Solution Accelerator Model Scoring

 

5. Automation

 

In order to automate the solution, the training and scoring pipelines must be published and a PipelineEndPoint must be created. Once that’s done, the PipelineEndpoint can then be invoked from Azure Data Factory. Specifically, the Azure Machine Learning Pipeline Activity is used. Note that the training and scoring pipelines can be collapsed into one pipelines if the training and scoring occur consecutively.  

 

 

Next Steps

 

Azure Machine Learning Documentation 

Many Models Solution Accelerator

Many Models Solution Accelerator Video

Azure Data Factory: Azure Machine Learning Pipeline Activity

MLOnAzure GitHub: Getting started with Pipelines

MLonAzure Blog: Getting Started with Azure Machine Learning for the Data Scientist

MLonAzure Blog: Azure Machine Learning service Pipelines

MLonAzureBlog: Azure Machine Learning Pipeline - ParallelRunStep

MLonAzureBlog: Video Walkthrough of ParallelRunStep

LinkedIn Group: Machine Learning on Azure

 

 

1 Comment
Microsoft

Thanks for sharing!